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1.0  INTPODUCTION  Mfi  SOMHRRY 


The  Air  Force's  approach  to  solving  the  computer  proliferation  problem 
by  specification  of  a  standard  instruction  set  architecture  has  great 
potential  for  reducing  software  development  and  maintenance  costs 
while  simultaneously  capturing  the  bene^ts  of  technology  infusion. 
Realization  of  this  potential  can  only  be  achieved,  however,  if 
different  implementations  of  HIL-STD-1750  do,  in  fact,  precisely 
represent  the  architecture.  The  importance  of  rigorous  verification 
that  these  representations  are  correct  cannot  be  overstated.  For 
these  reasons,  IBM  considers  the  successful  implementation  of  a 
MIL-STD-1750  certification  facility  to  be  a  critically  important 
element  of  the  Air  Force's  computer  architecture  standardization 
effort.  This  study  has  been  undertaken  to  assist  in  the 
i up le mentation  of  that  facility. 

This  study's  purposes  and  goals  follow: 

•  Investigate  methods  of  verifying  computers  to  an  Instruction 
Set  Architecture  and  recommend  a  method  suitable  to  SEAFAC 
for  certifying  vendor  produced  implementations  of 
MIL-STD-1750. 

•  Make  cost-effective  use  of  existing  SEAFAC  resources. 

•  Strive  for  vendor/implementation  independence. 

•  Recommend  a  verification  approach  based  on  a  cost  trade-off 
analysis. 

•  Provide  sufficient  descriptive  information  about  the 

recommended  approach  so  that  the  Air  Force  can: 

Plan  future  funding  and  personnel  requirements. 

-  Write  specifications  for  hardware  and  software 

necessary  to  support  approach. 

-  Contract  for  (or  develop  internally)  the  necessary 
facilities. 

The  Air  Force  has  provided  the  following  guidance  regarding  the 
certification  objectives: 

1.  The  Air  Force  desires  to  maintain  compatibility  of  support 
software  (this  implies  both  assembler  source  code 
compatibility  as  well  as  object  code  compatibility  in  what 
is  equivalent  to  the  problem  state). 

2.  The  approach  should  address  a  spectrum  of  computers  from  the 
microprocessor  to  the  high  performance  stand-alone  computer. 
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3.  The  Air  Force  desires  to  maintain  an  open  relationship  with 
vendors  with  regard  to  certification  plans,  programs  and 
procedures. 

4.  The  HIL-STD-1750  Instruction  Set  Architecture  User's  Group 
should  be  used  as  the  vehicle  for  effecting  changes  to  the 
standard. 

5.  From  a  practical  viewpoint,  the  certification  process  for 
each  machine  should  not  exceed  two  weeks,  based  on  an  eight 
hour  day. 

6.  The  Air  Force  does  not  desire  to  define  a  standard  AGE 
interface  as  an  additional  standard.  However,  it  would  be 
acceptable  to  require  seme  minimum  functional  capability,  a 
common  I/O  interface,  as  well  as  a  minimum  memory 
configuration  to  support  verification. 

7.  Only  one  vendor  will  be  certified  at  a  time. 

e.  Eecertif ication  is  not  planned.  However,  retesting  may  be 

considered  when  operational  problems  are  identified  or  when 
errors  or  updates  occur  in  the  certification  process. 

This  guidance  forms  a  basic  set  of  assumptions  used  in  this 

study.. 

This  study  investigates  verification  approaches.  IBM  has  conducted  a 
literature  search  to  identify  any  novel  verification  approaches  and  to 
gather  data  from  the  public  domain;  the  IBM  Federal  Systems  Divisior 
team  members  have  visited  other  IBM  locations  to  gather  data  about  the 
various  approaches  inside  the  corporation  and  have  consulted  with  the 
Owego  exnerts  concerned  with  verification. 

This  document  examines  the  following  verification  approaches: 

•  Acceptance  Test  Program 

AN/AYK-ISA 

ACAM 

-  User  Oriented  Micro  Processor 

•  Random  Instructions 

•  Analytical  Research 

•  Architectural  Verification  Program  -  System  360/370 

•  Diagnostic 
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•  Functional  Test  Program 

•  Instruction  Set  Processor  Specification 

•  Lockstep 

The  study  methodology  uses  the  following  3-step  approach: 

Step  1  Apply  pass/fail  criteria  (i.e. ,  feasibility)  to  each 

approach. 

Step  2  Measure  each  verification  approach  which  survives  the 
pass/fail  criteria  for  SEAFAC  impact,  time  to  perform  a 
verification  test,  non-recurring  start-up  costs, 
recurring  costs,  and  quality  of  the  approach. 

Step  3  Select  the  best  approach  under  which  SEAFAC  may 

implement  the  certification  capability. 

The  recommendation  of  this  study  is  a  certification  process  which 
invokes  two  verification  phases  to  provide  a  superior  means  of 
implementing  the  certification  facility.  The  first  phase  uses  a 
deterministic  verification  program  based  upon  modifications  to  the 
AN/AYK-ISA  ATP,  These  modifications  delete  non-MIL-STD-1750  features 
and  add  additional  tests  where  required.  It  is  followed  by  the  second 
phase  in  which  randomly  generated  sequences  of  instructions  are  run  on 
a  niL-STD-1750  simulator  and  the  machine  under  test:  the  results  are 
then  compared  to  determine  their  validity. 

The  time  estimate  required  to  implement  this  certification  process  at 
SEAFAC  is  a  time  period  of  a  year  and  a  half;  the  total  cost  estimate 
for  the  certification  process  is  $1,239,000  using  1979  dollars, 
$70,000  man  year  rate,  and  other  Air  Force  guidelines. 
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2.0  SCOPE  OF  STUDY 


The  objective  of  this  study  is  to  determine  the  methodologies  and 
identify  the  resources  required  to  provide  the  Air  Force  with  the 
capability  of  verifying  compliance  and  certifying  vendor  produced 
ccmputers  to  HTL-STD- 17 50.  This  study  identifies  plans,  procedures, 
hardware  support  items,  support  software  items,  interfaces,  personnel, 
and  other  resources  required  to  provide  ASC/ENASD  Systems  Engineering 
Avionics  facility  (SEAFAC)  with  the  capability  to  determine  computer 
instruction  set  compliance  to  MIL-STD-  1750. 

This  study  addresses  various  ootential  approaches  to  accomplish  the 
certification  process,  and  the  identification  of  hardware,  software, 
interface,  personnel,  and  miscellaneous  resources  required  to  support 
each  approach.  Each  approach  was  evaluated  considering  the  present 
capabilities  of  the  existing  SEAFAC  facility.  The  advantages  and 
disadvantages  of  each  approach,  and  their  cost  impacts  were  ’Iso 
considered. 


2.  1  '^"'UDY  ACTIVITIES 


The  following  summarizes  the  program  tasks  undertaken  during  this 
study: 

Ta?k  1.0  iientifi  Apprpaches;  Architecture  verification 
approaches  were  researched.  Approaches  used  from  inside  IBR  FSD 
and  IBB  commercial  were  surveyed;  the  Acceptance  Test  Plan  for 
the  AN/AYK-ISA  was  included  by  direction  of  the  Air  Force;  and  a 
survey  or  the  literature  describing  activities  in  the  computer 
industry  was  completed. 

Task  2.0  Analyze  Air  Forge  (SEAFACI  Resources:  This  task  was 
undertaken  to  understand  the  existing  SEAFAC  resources  and 
identify  any  newly  required  resources  based  on  the  verification 
approaches. 

Task  3.0  Trade-Off  Of  Approaches:  This  task  was  broken  into 
principally  two  sub-tasks.  In  the  first  sub-task  pass/fail 
criteria  (such  as  feasibility)  was  applied  to  each  of  the 
approaches.  The  approaches  which  passed  were  then  analyzed 
regarding  their  Non-Recurring  Start-Op  costs,  and  Recurring  costs 
in  order  to  select  the  best  verification  approach  or  combinations 
of  verification  approaches. 

TasK  4.0  Develop  Certification  Elan:  k  detailed  plan  for  the 
recommended  approach  was  developed  in  this  task  in  order  to  be 
used  as  a  guide  for  implementing  the  verification  capability. 
The  plan  includes  detailed  facility  requirements  and 
configurations,  a  detailed  mechan izaticn  plan  describing  how 
vendors  will  be  able  to  attain  certification,  suggested 
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organizational  structures  for  effective  control  of  the 
certification,  detailed  designs  and  procedures  to  test  each 
function,  and  impacts  to  HIL-STD-175C  which  pertained  to  its 
testability. 

Task  5.0  Produce  Study  Deliverables;  This  task  results  in  a 
draft  final  report,  a  final  briefing,  and  a  final  report  at 
completion  of  the  study  in  accordance  with  the  Statement  of  flork. 


2.2  PEBSPBCTIVE 


The  process  of  computer  architecture  verification,  which  is  extremely 
ccmplex  for  a  single  manufacturer,  is  further  complicated  for  the  Air 
force  because  the  Air  Force  must  deal  with  multiple  manufacturers  and 
formally  certify  that  a  machine  has  passed  the  architecture 
verification  process. 


2.2.1  Architecture  Verification 


The  task  of  architecture  verification  is  to  prove  that  a  given 
implementation  of  the  architecture  performs  all  functions  specified, 
and  executes  all  combinations  of  those  functions  with  clearly  defined 
results.  This  implies  not  only  obtaining  the  desired  results  but  also 
ensuring  that  no  other  conditions  except  those  specified  are 
performed. 

For  example,  MIL-STD- 1750  states  for  single  precision  compare; 

"The  single  precision  derived  operand,  DC,  is 
compared  to  the  contents  of  FA.  Then,  the 
condition  status,  CS,  is  set  based  on  whether  the 
contents  of  RA  is  less  than,  egual  to,  or  greater 
than  the  00.  The  contents  of  FA  are  unchanged." 

It  is  important  to  realize  that  statements  like  the  above  also  imply 
that  a  number  of  unstated  conditions  (e.g.,  other  status  bits, 
registers,  etc.)  are  unchanged.  Verifying  that  unrelated  conditions 
ate  noi  affected  is  as  important  as  verifying  expected  results. 

Items  to  be  checked  in  each  verification  test  include; 

The  function 
Data  patterns 

All  registers,  both  used  and  unused 
All  interrupts 

bain  Storage  locations,  both  used  and  related 
Condition  codes  set  or  unaffected 
Error  indicators 
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Order  of  occurrence  of  predicted  events 
Interference  due  to  siaultaneous  I/O  and/or  interrupts 

Each  architectural  verification  method  has  associated  with  it  a  degree 
of  quality,  which  is  a  measure  of  how  good  that  method  is  at  finding 
inconsistencies  between  the  hardware  implementation  and  the 
architecture  specification.  This  quality  may  be  viewed  from  three 


different 

perspectives: 

1. 

Completeness 

2. 

Coverage 

3. 

Confidence 

2.2.2  Certification 


The  act  of  conferring  certification  has  associated  with  it  certain 
liabilities.  It  is  important  that  the  study  consider  the  procedural 
issues  of  certification  so  that  these  liabilities  are  minimized.  Some 
examples  of  these  procedural  issues  follow. 

Bhile  the  ideal  goal  of  certification  is  to  ensure  100  percent 
compliance  with  the  specification,  experience  and  common  sense  suggest 
that. this  cannot  be  obtained  at  a  reasonable  cost,  (as  is  discussed 
later) .  As  a  result,  less  than  perfect  verification  must  be  accepted. 

It  is  likely  that  the  initial  certification  capability  which  is  put  in 
place  will  be  subject  to  improvements  over  time  as  experience  is 
gained.  Because  of  a  lack  of  maturity,  the  verification  process  as 
first  implemented  may  falsely  indicate  compliance  and  allow  a 
deviation  to  escape  detection.  Suppose  that  Machine  A,  which  has  been 
granted  certification,  is  later  retested  and  found  to  be 
non- representative  of  the  architecture.  It  is  recommended  that  a 
process  be  put  in  place  to  document  this  variance  and  publish  that 
information  to  any  potential  user. 

The  architecture  specification  itself  is  not  fixed  for  all  time. 
Father,  changes,  corrections,  and  extensions  will  be  made  over  time. 
Again  it  is  recommended  that  retesting  with  publication  of  the  results 
be  required  of  all  previously  certified  computers. 


2.2.3 


It  is  essential  to  have  a  complete,  detailed,  unambiguous 
documentation  of  the  architecture  to  ensure  that  a  compatible 
verification  is  possible  for  all  implementations.  This  document  must 
specify  the  functions  available  for  use  in  programming  the  computei 
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ircluding  all  significant  side  affects.  It  does  not  tell  the  hardware 
designer  what  technigues  or  technologies  to  use  in  his/her 
inplenentation  of  the  architecture.  It  nay  include  Inplenentation 
Notes  which  describe  the  intentions  behind  certain  operations  or 
functions  and  which  nay  aid  the  designer  in  any  hardware  trade-off 
decisions. 

Typically  this  document  includes: 

Processor  Structure 

Nain  Storage 
Addressable  Registers 
Instructions  and  Data  Format 
Addressing  Nodes 
Expanded  Addressing  Technique 
Nachine  Status 

Interrupt  System  and  Status  Switching 

Instruction  Repertoire 

Input/Output 

Channels 

Timers 

Discretes 

I/O  Interrupts 

1/0  Instruction  Repertoire 

Protection  Features 

Multiprocessing  Facilities 

Multiprogramming  Facilities 

Security  Features 

Alternative  Subset  Inplenentation 

Growth  Provisions 


2.2.4  Further  Refinements  to  MIL-STD- 1750 


""he  Statement  of  Work  in  the  Air  Force’s  PFP  indicates  that  the  study 
hould  be  concerned  with  areas  of  HIL-STD-1750  which  require  further 
efinition  because  of  testability  impacts.  Specifically,  "The 
ccntractor  ...  shall  identify  inoacts  on  HIL-STD-1750  which  pertain  to 
its  testability." 

*r  example  of  this  type  of  architectural  impact  due  to  testability 
considerations  arises  frcn  the  Lockstep  testing  approach,  which  will 
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t€  discussed  in  detail  in  a  later  section.  Basically,  t.his  nethod 
requires  that  the  computer  have  a  bit  in  its  Program  Status  Word  (?SW) 
which  causes  it  to  enter  a  trace  mode.  In  the  trace  mode,  an 
instruction  is  ejcecuted  and  then  the  trace  interrupt  occurs.  This 
trace  interrupt  saves  all  the  status  of  the  computer  in  a  status  area 
and  loads  a  new  PSH.  Having  a  copy  of  the  complete  status  of  the 
computer  at  the  completion  of  each  instruction  execution  permits  that 
status  to  be  compared  with  similar  status  from  a  certified  computer 
(that  is,  one  that  is  certified  as  a  member  of  the  faiiily)  .  Thus,  if 
the  Lockstep  testing  approach  is  chosen,  it  would  impact  the 
definition  of  MIL-STD-1750  by  recruiring  the  definition  of  a  trace 
mode. 

Specific  refinements  to  MIL-STD-1750  will  be  covered  in  the 
recommendation  section. 


2.2.5  Industry  Activities 


Computer  development  has  multiple  stages  in  which  computer 
verification  is  required; 

1.  As  a  bring-up  tool  by  Engineers. 

2.  As  a  verification  tool  by  Quality  Assurance. 

3.  As  a  final  box  checkout  by  Manufacturing. 

4.  As  a  diagnostic  check  by  Field  Engineering. 

Each  area  inside  IBM  was  surveyed  for  their  approach  to  verification 
and  how  this  could  be  applied  to  the  Air  Force's  problem. 


2.3  AIP  FORCE  RESPONSIBILITIES 


The  Air  Force  must  provide  complete,  detailed,  unambiguous 
documentation  of  the  architecture  (HIL-STD- 1750) .  This  is  required 
because  informal  communication  channels  are  difficult,  unreliable,  and 
unmanageable  within  one  manufacturer,  and  almost  impossible  across  a 
variety  of  manufacturers.  For  example,  a  "LOAD  BYTE"  instruction 
could  indicate  that  a  8-bit  byte  operand  from  main  storage  is  loaded 
into  a  16-bit  general  register;  but  may  neglect  to  specify  what 
happens  to  the  other  8-bits  in  that  general  register.  The  consequence 
is  that  one  implementation  might  clear  the  16-bit  general  register 
before  loading  the  byte  while  another  might  insert  the  byte.  Since 
the  verification  program  cannot  test  outside  of  the  specification,  the 
two  implementations  pass  certification  and  a  programmer  moving  from 
one  implementation  to  another  may  have  a  very  subtle  error  to  uncover 
due  to  this  difference. 
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The  Air  Force  should  extensively  test  for  compatibility  of  a  vendor's 
ittplementation  to  the  MIL-STD-1750  specification.  It  is  better  to  err 
on  the  side  of  conservatism  than  to  under  test  an  implementation. 
Additionally,  the  Air  Force  must  be  careful  not  to  test  outside  the 
MIL-STD-1750  specification  as  indicated  previously. 

The  Air  Force  should  encourage  vendors  to  have  their  implementations 
certified.  In  order  to  do  this,  the  certification  process  should  be 
as  painless  as  possible.  For  example,  the  Air  Force  should  freely 
make  available  copies  of  the  verification  programs  for  the  vendors  to 
"pre-test"  their  implementations.  Furthermore,  when  the  certification 
process  detects  a  problem,  the  Air  Force  should  provide  as  much 
information  as  possible  about  that  failure,  and  then  should  resume 
testing  for  other  errors  in  order  to  provide  the  vendor  with  as  much 
information  as  feasible  about  their  computer.  (However,  it  must  be 
noted  that  the  information  provided  need  not  be  to  the  extent  that  the 
Air  Force  is  debugging  the  vendor  machine.) 

The  Air  Force  must  maintain  full  documentation  of  the  certification 
process  in  order  to  duplicate  the  certification  process  at  a  later 
point  in  time  or  to  reverify  that  a  certification  was  properly 
undertaken.  When  retesting  is  indicated  (as  described  in  another 
section) ,  the  Air  Force  must  document  and  publish  the  results- 


2.4  STUDY  GBOUND  fiULES 


The  results  of  this  study  are  based  upon  the  set  of  ground  rules 
(developed  with  Air  Force  guidance)  which  follow: 

1.  In  SEAFAC,  all  hardware  facilities  (in  particular  the  VAX 
11/780  computer,  the  PDF  11/55  computer,  and  the 
MIL-STD-1553  interface  to  these  computers)  should  be 
considered  zero  cost  ftcm  both  a  Non-Recurring  Start-Up 
(NESO)  and  a  Recurring  basis. 

2.  The  certification  process  should  take  less  than  two  weeks. 
A  manned  work  week  is  comprised  of  five  8-hour  days; 
however,  an  automated  process  without  manual  intervention 
could  be  available  for  second  and  third  shifts. 

3.  Only  one  vendor  at  a  time  would  be  certified  in  the  SEAFAC 
facility  due  to  problems  associated  with  proprietary 
hardware/information  about  vendor's  competitors. 

4.  It  is  better  to  be  conservative  and  over  test  than  to  under 
test  during  the  certification  process. 

5.  In  order  to  be  certified,  a  minimum  storage  configuration 
(like  64K)  and  a  standard  I/O  channel  (like  MIL-STD-1553  or 
ES-232)  could  be  reouired. 
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6.  A  standard  Ground  Support  Equipment  (GSE)  interface  is  not 
required  because  a  standard  Ground  Support  Equipment 
interface  for  a  family  of  computers  starting  with  a  one  page 
micro  processor  (with  HAH)  embedded  in  a  system  up  to  a 
standalone  computer  in  its  own  box  could  unfairly  penalize 
the  former  class. 

7.  For  converting  man  years  into  cost,  a  1979  labor  rate  of 
$70,000  per  professional  Air  Force  employee  should  be  used 
which  is  also  approximately  the  same  for  a  professional 
industry  employee. 

8.  During  the  ten  year  expected  life  for  the  MIL-STD- 1 750 
architecture , 

a.  from  313. 2K  to  522K  source  lines  of  assembly  code  are 
expected  for  operational  programs,  and 

b.  twenty  computers  are  expected  to  be  submitted  for 
certification  by  SEAFAC  and  ten  of  these  would  be 
resubmitted  after  initially  failing. 

9.  Programmer  productivity  is  75  lines  of  assembler  source  code 
per  man  month  and  110  lines  of  Higher  Order  Language  (HOL) 
source  statement  per  man  month  for  operational  programs. 
Parenthetically,  a  1  to  3  factor  is  applied  for  expanding 
from  one  HOL  statement  into  assembler  code  statements. 
Software  maintenance  costs  are  based  on: 

a.  the  existence  of  two  errors  per  one  thousand  assembler 
source  lines  of  delivered  code,  and 

b.  each  error  requires  one  man  week  to  repair. 
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3 . 0  DEFINITIONS 


This  section  provides  definitions  for  a  numter  of  concepts  which  are 
used  throughout  this  report.  Definitions  are  provided  here  because 
the  terms  they  define  are  used  quite  loosely  in  the  industry.  In 
addition  to  the  definitions,  comments  are  provided  which  relate  the 
terms  to  this  study. 


3.1  CCNPLETENESS 


Completeness  is  a  measure  of  how  thorough  an  architectural 
verification  program  is  in  testing  that  a  particular  machine  meets  an 
architecture  specification.  A  comnlete  verification  of  an 
architecture  would  require  checking  all  possible  combinations  of 
lernorv  locations  using  all  possible  combinations  of  instructions  and 
all  possible  combinations  of  data  patterns,  as  well  as  checking  all 
conditions  which  are  to  remain  undisturbed.  The  measure  could  be 
expressed  by  the  number  of  combinations  tested  as  a  percentage  of  the 
number  of  combinations  possible.  This  is  not  a  very  practical  measure 
to  acply  to  architectural  verification  programs,  however,  since  a 
complete  verification  of  an  architecture  such  as  MIl-STD-1750  would 
take  an  unreasonable  amount  of  time.  For  example,  just  to  check  all 
possible  pairs  of  instructions  with  each  memory  location  and  each  data 
pattern  would  require  ten  years  (assuming  a  two  microsecond 
instruction  time). 


(1952  pairs  r  2^*  memory  locations  x  data  patterns  x 
2  X  10-*  seconds  =  326  ,632,  262.  9  seconds)  >  10  years. 


Obviously,  exhaustive  testing  for  a  complete  verification  is  not 
practical. 

For  most  verification  tests,  the  completeness  percentage  would  be  a 
figure  near  zero.  Also,  verification  Method  A  could  have  one  hundred 
times  the  completeness  of  verification  Method  B,  while  A  only 
exercised  half  of  the  instructions  and  B  exercised  all  of  them.  For 
these  reasons,  completeness  would  not  be  a  useful  measure  for 
distinguishing  between  architectural  verification  programs. 


3.2  COVERAGE 


Coverage  is  the  percentage  of  the  computer  hardware  (ir  terms  of 
gates,  wires,  microcode,  memory  locations,  etc.)  which  is  tested  by  an 
architectural  verification  program.  This  term  is  commonly  applied  to 
the  microcode  portion  of  the  hardware.  A  measurement  cf  coverage  has 
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t€€n  nade  for  several  architectural  verification  programs.  See,  for 
example.  Criteria  for  Architecture  Verification,  by  E.  C.  Varney  and 
K.  P.  Groundwater-  Coverage  for  developing  a  computer  (as  applied  to 
microcode)  is  a  useful  concept,  since  less  than  100  percent  coverage 
implies  that  parts  of  the  machine  remain  untested  (which  may  contain 
errors) .  When  applied  to  architectural  verification,  however, 
coverage  has  two  limitations.  First,  measurement  of  the  coverage  of 
gates  and  wires  is  very  difficult.  This  is  significant  since  failures 
of  gates  and  wires  contribute  to  architectural  discrepancies  as  surely 
as  do  microcode  faults.  Second,  and  more  important,  attainment  of  100 
percent  coverage  does  not  guarantee  that  no  architectural 
discrepancies  remain  in  the  machine.  Simply  exercising  each  location 
cf  microcode  does  not  mean  that  every  execution  path  has  been  taken. 
Furthermore,  coverage  is  net  expected  tc  be  a  goed  measure  for 
distinguishing  between  architectural  verification  methods,  since  most 
methods  are  easily  capable  of  achieving  100  percent  or  nearly  100 
percent  coverage  in  practice. 


3.3  CCNFIDENCE 


Ccnfidence  is  the  degree  of  certainty  that  a  given  seftware  module 
(which  is  known  to  be  correct)  will  execute  properly  on  a  machine 
which  has  been  certified  through  the  use  of  an  architectural 
verification  program.  While  confidence  would  be  a  useful  measure  for 
distinguishing  between  architectural  verification  methods,  there  is  no 
simple  means  of  obtaining  estimates  of  the  confidence  associated  with 
each  architectural  verification  method. 


3.4  CnALITY 


Quality  is  defined  as  the  degree  to  which  an  architectural 
verification  method  is  capable  of  determining  compliance  of  a  hardware 
implementation  to  an  architectural  specification-  It  is  measured  by 
the  projected  total  number  of  architectural  discrepancies  expected  to 
remain  in  a  machine  after  verification.  This  concept  is  applied  in 
this  study  and  is  discussed  in  detail  in  Section  5. 
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3.5  SECOND-OfiDES  EFFECTS 


S€cond-order  effects  are  program  executicr  errors  which  are  the  result 
of  the  icteractions  between  two  instructions.  For  example,  the  ADD 
and  HDLTIPLY  instruction  could  function  properly  when  executed 
independently,  but  fail  when  executed  sequentially.  Second-order 
effects  are  both  very  real  and  very  hard  to  find.  The  definition  of  a 
completeness  suggests  the  difficulty  in  testing  all  instruction  pairs. 
Cfcviously,  even  higher-order  effects  are  possible. 


3.6  ACCEPTANCE  TEST  PROGRAM 


The  Acceptance  Test  Program  (ATP)  is  the  software  portion  of  the 
Acceptance  Test  Procedure  which  is  used  to  sell-off  the  hardware, 
his  program  attempts  to  verify  that  all  functions  defined  in  the 
ardware  specification,  which  can  be  observed  by  the  programmer, 
perform  properly.  An  ATP  usually  contains  verification  tests, 
performance  tests  and  sections  used  for  detailed  measurements. 


3.7  APCHITECTURAL  VERIFICATICN  PPCGFAM 


An  Architectural  Verification  Program  (AVP)  is  a  computer  program  used 
to  demonstrate  that  a  specific  hardware  implementation  of  a  computer 
architecture  performs  all  programmer  observable  functions  as  defined 
in  the  architecture.  All  basic  architectural  features  are  assumed  to 
function  properly  and  are  used  freely  throughout  testing.  An  AVP 
usually  not  only  verifies  that  what  is  expected  is  performed,  but  also 
that  no  unintended  functions  are  performed. 


3.8  DIAGNOSTIC  PROGRAM 


A  Diagnostic  Program  is  a  computer  program  written  to  verify  the 
proper  operation  of  the  hardware  and  to  attempt  to  isolate  any 
failures  (typically,  the  Shop  Replaceable  Dnit  level) .  The  size  of 
the  program  is  reduced  by  knowledge  of  the  hardware  implementation  and 
by  the  addition  of  hardware  and/or  microcode  to  assist  it  in  achieving 
its  objectives.  The  program  is  highly  implementation  dependent. 
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3.9  FUNCTIONAL  TEST  PBOGBAM 


A  Functional  Test  Program  (FTP)  is  a  computer  program  which  is  used 
for  initial  debug  of  hardware.  It  tests  all  functions  of  the  hardware 
clservable  to  the  programmer.  An  FTP  is  written  under  the  premise 
that  all  architectural  features  require  testing  prior  to  use.  It  may 
take  advantage  of  the  hardware  implementation  to  reduce  the  size  of 
the  program. 


3.10  COWPDTEE  HARDWARE  DESCRIPTION  LANGUAGES  (CEDI) 


CEDIs  provide  for  precise  syntactical  descriptions  of  the  functional 
behavior  of  digital  circuits.  They  are  used  to  describe  the  logic 
gate  networks,  sequential  circuits,  modules,  their  connections,  and 
their  control  in  a  digital  system. 


Examples:  Instruction  Set  Processor  (ISP)  ; 

Used  in  ISP  Verification  Approach 
Language  for  Symbolic  Simulation  (LSS) ; 
Used  in  Analytic  Verification  Approach 


3.11  CERTIFICATION  PROCESS  VERSUS  VERIFICATION  PRCGEAH 


Certification  is  the  process  of  testing  a  computer  by  using  a 
verification  program  in  order  to  determine  that  the  computer  conforms 
(or  fails)  to  an  architectural  specification  like  flIL-STD-1750. 
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U . 0  STUDY  aETHODOLOGY 


This  section  provides  a  discussion  of  the  reasons  behind  the  selection 
of  the  cost  model  approach  and  provides  the  rationale  for  the 
elimination  of  other  approaches-  It  describes  the  criteria  that  are 
used  to  evaluate  the  various  architecture  verification  methods  and 
presents  a  systematic  approach  that  is  used  to  trade-off  those 
methods.  This  section  also  describes  the  methodology  that  was  applied 
in  this  study  for  data  gathering  and  data  analysis. 


h.1  GOALS  OF  THIS  STUDY 


In  lEM's  proposal  to  the  Air  Force,  a  number  of  verification  methods 
were  discussed.  These  approaches  were: 

1.  Acceptance  Test  Procedures  for  Existing  IlIL-STE-1750  Related 
programs  (AN/AYK-15A,  ADAM,  User  Oriented  Micro  Processor) 

2.  Random  Instructions 

3.  Analytical  Research 

4.  Architectural  Verification  Program  -  System  26C/37C 

5.  Diagnostic 

6.  Functional  Test  Program 

7.  ISP 

8.  Lockstep 

The  purpose  of  this  study  is  to  determine  the  advantages  and 
disadvantages  of  each  of  these  verificaticn  methods  (in  light  of 
SEAFAC  resources) ,  to  systematically  evaluate  these  advantages  and 
disadvantages  to  select  the  best  alternative,  and  finally,  to  identify 
the  detailed  designs  and  procedures  needed  to  equip  SEAFAC  with  the 
capability  of  determining  whether  or  not  any  machine  meets  the 
reguirements  of  IlIL-STD-1750.  Sufficient  descriptive  information  is 
provided  on  the  implementation  of  the  recommended  approach  so  that  the 
Air  Force  can:  plan  future  funding  and  personnel  reguirements,  write 
specifications  for  the  hardware  and  software  necessary  to  support  the 
approach,  and  contract  for  (or  develop  internally)  the  necessary 
facilities.  A  summary  of  the  study  approach  is  shown  graphically  in 
Figure  4-1.  The  verification  approaches  are  evaluated  against  the 
fcllcwing  criteria: 

Impact  to  SEAFAC  Resources  -  necessary  additions,  such  as 
special  test  equipment  or  specially  trained  personnel  must 
be  viewed  as  a  cost. 
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EACH  CERTIFICATION  APPROACH 
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Figure  4-1. 


Summary  of  Study  Approach 
(Sheet  1  of  1) 
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2.  Tiaie  Required  to  Perform  Validation  Effort  -  especially 

lengthy  procedures  invoke  labor  costs  and  raocopoiize 

valuable  test  resources. 

3.  Non-fiecurring  Start-Op  (NPSO)  Costs  -  includes  items  such  as 
writing  architectural  verification  software  and  test 
procedures. 

y.  Recurring  Costs  -  includes  items  such  as  architectural 
verification  software  maintenance,  and  staffing  during  the 
certification  process.  This  figure  is  developed  as  the  cost 
per  certif icat ion- 

=.  Vendor  Independence  -  the  verification  method  under 
consideration  must  not  discriminate  cn  the  basis  of 

vendor-unique  items  which  are  not  specified  by  the 

MIL-STD- 175.0  architecture.  Per  example,  a  method  which 
requires  that  a  particular  type  Ground  Support  Equipment 
interface  be  present  to  perform  the  test  sacrifices  vendor 
independence. 

6-  Application  Independence  -  the  verificaticn  metnod  under 
consideration  must  not  discriminate  on  the  basis  of  items 
which  are  dictated  by  the  application  requirements,  but 
which  are  not  specified  by  the  architecture ,  e.g.,  machine 

weight,  power,  volume,  or  coding  requirements. 

7.  Quality  -  the  degree  to  which  the  method  verifies  that  the 
hardware  actually  reflects  the  architecture.  The  quality 
criteria  encompasses  the  notions  cf  Completeness,  Coverage, 
and  Confidence  defined  earlier.  These  concepts  are 
discussed  in  detail  in  Section  5. 

8.  Cost  of  Software  Revalidation  -  if  a  validation  method  is  of 
less  than  perfect  quality,  there  is  some  finite  probability 
that  any  software  developed  in  consonance  with  tlIL-SID-1750, 
even  software  previously  validated  on  a  certified  machine, 
can  fail  on  a  second  certified  machine  used  for  a  new 
application.  It  would  therefore  be  necessary  co  revalidate 
any  code  which  will  be  used  on  this  second  machine. 

After  evaluating  the  different  verification  methods  against  these 
criteria  (the  methodology  of  which  will  be  discussed  later)  ,  the  next 
step  is  to  systematically  analyze  the  results  and  select  the  best 
method  for  SEAFAC. 


y.2  THE  THADE-OPF  APPROACH 


The  following  evaluation  method  is  employed  in  this  study.  First,  the 
VENDOR  INDEPENDENCE  and  APPLICATION  INDEPENDENCE  criteria  are  treated 
in  a  oass/fail  manner.  Because  it  is  essential  that  the  verification 
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■ethcd  enployed  test  any  inpleaentation  of  HIL-STD- 1750,  any  nethod 
which  fails  these  criteria  is  rejected  and  is  not  subjected  to  further 
consideration. 

The  reaaining  criteria,  SEAPAC  IMPACT,  TIME,  NEED,  EECDEEING  ZLEMEKTS, 
COALITT,  and  SOFTiABE  VALIDATION  ELEMENTS,  are  converted  to  costs. 
(Mow  this  is  accomplished  is  discussed  in  Section  5.)  Each 
verification  method  that  survives  the  pass/fail  criteria  is  measured 
against  these  criteria.  Since  each  criteria  represents  a  cost,  a 
total  cost  is  obtained  for  each  method.  The  method  vith  the  lowest 
cost  is  selected  as  the  best. 

Nhile  some  of  these  criteria  are  easily  recognized  to  be  of  a  cost 
nature,  others  do  not  have  an  obvious  cost  relationship.  Therefore, 
each  of  these  criteria,  and  the  techniques  that  will  be  used  to 
measure  them,  will  be  discussed  in  detail. 


4.3  CATA  COLLECTION 


The  first  step  in  the  study  is  to  collect  data  cn  all  of  the 
verification  approaches.  This  is  necessary  to  fully  understand  each 
of  the  approaches  and  to  support  the  cost  model. 

Data  collection  was  accomplished  in  five  phases;  literature  search, 
interviews  and  site  visits,  ECs  and  Validation  costs,  local  cost 
estimaticn,  and  miscellaneous  trips. 

Dse  of  the  cost  model  required  that  the  following  types  of  data  be 
gathered  for  each  of  the  verification  approaches; 


Pass/Fail  Data 

•  vendor  independence 

•  application  independence 

•  feasibility 

•  uniqueness  from  other  approaches 

•  availability  of  information 

•  testable  within  a  two  week  period 

Cost  Data 

•  non-recurring  start-up  costs 

•  recurring  costs 

•  SEAFAC  resources 

Quality  Data 

•  Engineering  Changes  (ECs) 

•  Software  validation  costs 


(The  reasons  for  collecting  the  quality  data  are  discussed  in  Section 
5.)  In  addition,  detailed  descriptive  information  about  each  of  the 
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verificaticn  approaches  is  required. 


« . 3 . 1  Literature  Search 


An  intense  literature  search  has  been  Bade  to  support  the  researc.. 
being  done  on  the  MIL-STD- 1750  Certification  Study.  This  research 
crovided  additional  data  regarding  the  proposed  approaches;  but  did 
net  uncover  any  new  verification  approaches  to  be  in vesti^atea.  Tne 
fcllowing  literature  data  bases  were  Queried  by  IBM  Technical 
Information  Retrieval  Center: 


NTIS 

INSPEC 

IBM 

EIXF 


National  Technical  Information  Service 

Information  Service  in  Physics  Electrotechnology 
and  Control 

IBM  Internal  Technical  Feports/Eesearc h  Reports 
Engineering  Index 


The  results  of  this  search  can  be  found  in  the  Bibliography.  This 
lists  the  papers  and  articles  directly  relevant  to  this  study. 


4.3.2  Interviews  and  Site  Visits 


All  of  the  verification  approaches  investigated  (except  MIL-STD-1750 
ATPs)  have  been  used  within  the  various  IBM  Divisions.  To  obtain  the 
detailed  information  required,  the  Divisions  shown  in  Figure  4-2  were 
consulted.  Users  of  each  of  the  appreaches  were  icterviewed  and, 
where  possible,  the  originators  of  the  concepts  were  interviewed. 
(This  was  the  case  for  the  Lockstep,  Random,  and  Analytical 
approaches.)  Site  visits  were  made  to  the  T.  J.  Ratson  Research 
Center  at  Yorktown,  N.Y.  ,  the  System  Products  Division  at 
Feughkeepsie,  N.Y.,  the  System  Products  Division  at  Endicott,  N.Y., 
and  the  General  Systems  Division  at  Boca  Raton,  FLA. 


4.3.3  Engineering  Change  and  Software  Validation  Cost  Data  collection 


Engineering  Change  data  were  gathered  with  help  of  the  Owego  Part 
Number  Identification  User  System  (CPIDS).  This  automated  system 
permits  a  user,  via  a  display  terminal,  to  obtain  Engineering  Change 
and  part  number  information  for  the  avionics  computers  built  by  IBM  in 
Owego,  N.Y.  Engineering  Changes  of  interest  were  physically  pulled 
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IBM  DIVISIONS  CONSULTED 


Figure  U-2.  IBK  Eivisions  Consulted 
(Sheet  1  cf  1) 
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frcm  storage  and  scrutinized  to  detemine  their  applicability  to  the 
study. 

Scftware  validation  cost  information  «as  obtained  by  interviews  with 
the  software  personnel  responsible  for  the  validation  efforts.  These 
personnel  were  located  in  the  Air  Force,  the  Navy,  and  lEh. 


.  3 .  U  Internal  Da^  Gathering 


Tc  supplement  the  cost  data  gathered  through  the  fact  finding  trios 
and  extensive  literature  search,  various  lEM  -  Owego  sources  were 
utilized.  FSD  personnel  experienced  in  support  software  development, 
systems  integration,  and  hardware  testing  and  verification  contributed 
i;cst  data  and  other  infermatien  in  their  area  of  expertise.  This  data 
is  reflected  in  developing  the  cost  figures  related  tc  the  various 
approaches  described  in  Section  7. 


'J .  3 .  :*isceilaneous  Trips 


1 wc  other  trips  were  taken  by  tea®  members  in  support  of  this  study. 
The  first  was  to  3EAPAC  to  obtain  informaticn  about  the  equipment, 
personnel,  and  facilities  available  tc  support  the  aiL-STD-1750 
certification  effort.  The  second  trip  was  to  Palo  Altc,  California  to 
attend  thi  1979  International  Syspesiu®  cn  Cemputer  Hardware 
Eescripticn  Languages  and  their  applications.  This  allowed  the  latest 
available  data  on  tne  CRDL's  approach  to  be  included  in  this  study. 


U.4  TATA  ANALYSIS 


Analysis  of  the  gathered  data  began  with  the  application  of  t re 
pass/fail  criteria.  Additional  criteria  tr  these  discusseu  in  tne 
ptcpcsal  were  identified.  (The  cost  model  was  then  applied  to  those 
verification  approaches  which  survived  the  pass/fail  test.) 

The  Engineering  Change  data  and  software  validaticn  cost  aata  were 
analyzed  to  determine  the  cost  penalties  which  will  be  incurred 
because  c£  the  necessity  of  using  a  ncn-cerfect  certification  aetr.od. 
This  information,  combined  with  the  cost  data  above,  fotm  the  lasis 
for  the  study  recommendation. 
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5.0  COST  MODEL 


To  summarize  the  aaalysis  described  in  Section  4,  three  things  must  be 
dene  to  select  the  best  verification  method: 

1.  Apply  the  pass/fail  criteria, 

2.  Estimate  the  costs  for  the  remaining  approaches,  and 

3.  Treat  in  an  ad  hoc  manner  any  newly  identified  criteria 
which  do  not  fit  into  either  (1)  cr  (2). 

This  section  describes,  in  seme  detail,  the  concepts  behind  the  cost 
estimaticn  process  (item  (2)).  The  application  of  ttese  concepts  to 
the  gathered  data  is  discussed  in  Sections  6  and  7. 

This  section  is  nroken  into  two  parts.  The  first  part  discusses  the 
fcllcwinq  items:  SEAFAC  IMPACT,  TIME,  NFSU,  and  FECUBRING  COST. 
These  items  are  relatively  easily  converted  to  dell  a r  expenditures, 
which.  are  necessary  for  SEAFAC  to  establish,  use,  and  maintain  a 
certification  facility  for  MI L-STD- 1 75 0. 

Tie  second  part  of  this  section  discusses  ..DALITh  and  bCFTwAFZ 
VALIEATTCN  COSTS.  These  i terns  represent  costs  which  will  oe  incurred 
ever  the  lifetime  of  MIL-STL-1750  and  which  are  in  the  form  of  a 
penalty  which  will  be  paid  by  the  individual  ptectams  wcicn  use 
MIl-STD- 1 750.  They  are  not  direct  costs  tc  SEAFAC,  but  are  true  costs 
to  the  Air  force,  nonetheless.  Conversicn  cf  COALITT  tc  cost  is  more 
involved  than  the  conversion  of  the  ether  items  to  cost.  As  a  result, 
a  significant  amount  of  this  section  is  deveted  tc  discussion  of  the 
CCALITT  concepts. 


•=  .  1  COSTS  TO  SEAFAC 


■’his  s«^cticn  descrines  the  evaluation  criteria  SFAFAC  IMPACT,  TIME, 
NFS'’,  and  RECURRING  COST.  These  itcis  w:il  be  a  direct  cost  t  j 
'^FAFAC.  Thfc  descripticns  previlei  here  ar^  in  the  fctir  or  a  £Liori 
eicec*-icns;  tney  ace  included  here  as  exenrlary  information  tor  the 
rest  model.  Again,  the  actual  findings  of  the  study  appear  in 
Sections  n  and  7. 


'  .  1 .  ■*  I  ipdct  ty  SEAf  AC  fieso  u  rces 

FAFA  '-:ntdins  a  ,'C‘-‘at  deal  ;f  reEour''ef,  in<  luerne  turn  ev,i..fmtent 
an’  r  f  r  SOI,  ;.fc  I ,  wt.  ich  could  roeentially  I isetul  fci  ar  .Mt  ?i.  :t  ur 
Vi-r  1  f  >'3  iv^n.  .lowever,  a  rartirular  ve  1 1  h  ica  t  ion  method  iiint 

rtouire,  rot  examp.e,  seme  ur.irie  ♦est  *^oi!;rient  rc  rnteitdci-  wit:  n  t  e 
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utit  under  test.  A  cost  would  be  incurred  for  procuring  that 
equipuent  and  training  personnel  to  use  it- 

Arcther  verification  method  might  require  that  SIAFAC  hire  an 
architecture  expert.  A  cost  would  be  incurred  to  hire  this  person, 
pay  his  salary,  and  pay  his  benefits. 

Each  verification  method  is  examined  for  any  such  unique 
considerations. 


'.1.2  lime  Required  to  Perform  Validation  j 

The  time  required  to  perform  the  certification/verification  approach 
translates  into  a  cost  for  personnel  and  equipment  (required  to 
support  the  rate  of  testing  and  maintenance) . 


'.1.?  Non-Recurring  Start-Dp  Costs 


fany  tyres  or  one  time  start-up  costs  night  be  incurred  with  the 
various  vetirication  test  methods.  It  is  possible  that  a  particular 
method  woula  require  that  some  form  cf  architecture  verification 
program  either  be  written  or  modified.  Detailed  test  procedures  would 
likely  have  to  be  written  for  any  approach.  For  the  Lockstep 
approach,  it  may  oe  necessary  to  procure  a  known,  standard  machine. 
The  ISP  generated  AVP  method  may  reouire  development  cf  AVP  generation 
software.  Ine  Diagnostic  approach  might  require  analysis  of  each 
manufacturer's  programs.  Another  requirement  may  be  some  type  of 
Ground  Support  Equipment  (GSE)  or  1/C  interface  to  connect  a  unit 
under  test  to  tne  test  hardware. 


' . 1 . u  Recur  ting  costs 


"Software 

arrli<=5- 

r  1  a  t  f  ^ 
r  \  f  to  * 


tiiai  IS  to  he  written  must  also  be  maintained.  The  same 
t  ai.y  narawart  to  be  procured.  These  costs  are,  of  course, 
tf’  t.'.cir  j1  lifetime,  Pecurrira  costs  are  also  incurred 
ht  iitatting  Gurinc  the  validation  process. 


.'DALIIY  AND  VALIDATICN  CCSTS 

t  15  ipoortant  to  recognize  the  significance  of  the  quality  measure 
rf  arv  a  tc  r.i  tec  t  ut  e  verification  method  because  no  method  of  testing  a 
lacfin.t  wii.  ne  perfect,  anl,  as  a  result,  costs  will  be  incurred. 
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Quality  is  defined  as  the  degree  tc  which  an  architectural 
verification  method  is  capable  of  determining  compliance  of  a  hardware 
iapleraentation  to  an  architectural  specification.  The  unit  of  measure 
is  the  expected  number  of  architectural  discrepancies  remaining  in  a 
machine  after  architecture  verification.  The  quality  of  a  given 
verification  method,  then  is  an  assessment  cf  how  good  that  method  is 
at  finding  machine  defects  or  architectural  discrepancies. 

The  concept  of  quality  is  related  to  three  separate  concepts  which  are 
widely  used  in  the  industry.  These  are  ccmpleteness ,  confidence  and 
coverage.  Completeness  requires  checking  the  machine  for  all  possible 
data  tyres  with  ail  possible  instruction  sequences  in  all  possible 
memory  locations,  etc.  Hardware  coverage  is  the  percentage  of  a 
machine  that  is  tested  in  terms  of  gates,  memory  locations,  microcode, 
wires,  etc-  Confidence  is  the  degree  cf  certainty  that  a  software 
module  will  execute  properly  on  a  verified  machine.  These  concepts 
have  been  briefly  described  here  because  they  are  commcnly  referred  to 
in  industry.  None  of  these  three  concepts  are  used  in  the  cost  model, 
however,  for  reasons  cf  practicality  (discussed  in  Section  3) . 
Father,  the  concept  of  quality  will  be  used, 

Cne  would  expect  that  the  direct  cost  of  implementing  a  verification 
method  would  be  roughly  related  to  the  quality  of  that  method  as  shown 
in  Figure  5-1.  A  test  that  was  not  very  extensive  (and,  hence,  not  of 
high  quality)  would  not  be  difficult  tc  program  and  would  not  take 
long  to  cun,  making  it  low  in  cost. 


Cort  of  Verifying 
That  a  IVIachine 
Meets  the 
Architectural 
Specification 


Perfect 

Quality 

Figure  5-1.  Verification  Cost  as  a  Function  of  Quality  of  Test 
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Slight  additional  investment  to  this  iriniinal  test  nould  result  in 
large  gains  in  quality  as  more  and  more  instruction  types  and  formats 
vere  tested.  The  cost  of  approaching  perfect  quality,  however,  would 
be  extremely  large,  because  it  would  be  necessary  to  test  ail  possible 
data  combinations  with  all  possible  instruction  sequence  combinations 
in  all  possible  memory  locations,  etc. 


5.2.1  Beasons  for  Considering  Quality 


Achievement  of  perfect  quality  in  a  verification  method  would  require 
that  method  performs  exhaustive  testing.  That  is  not  feasible  for 
KIL-STD-1750  certification  (see  the  definition  of  Completeness). 

Since  any  useable  verification  test  method  is  of  less  than  perfect 
quality,  there  is  a  very  real  possibility  that  the  tested  machine  will 
still  have  undiscovered  architectural  discrepancies,  and  that  any 
operational  software  will  not  execute  correctly  (that  is,  as  defined 
in  the  architectural  specification  manual).  It  is  necessary, 
therefore,  to  check  all  software  that  is  to  run  on  that  machine  for 
proper  execution,  i.e.,  to  validate  that  software.  Typically,  this 
validation  is  initiated  in  a  laboratory  simulation  and  then  completed 
it  actual  flight  test. 

The  software  to  be  validated  can  be  broken  up  into  two  classes: 
existing  software  that  is  to  be  captured  for  this  machine  (e.g., 
navigation  modules,  executives),  and  new  software  that  is  to  be 
written  for  this  machine. 

The  cost  associated  with  revalidating  the  existing  software  (which  had 
already  been  validated  for  some  other  machine)  is  a  direct  result  of 
the  less  than  perfect  quality  of  the  verification  method.  If  a 
verification  method  were  100  percent  complete  in  certifying  that  a 
machine  met  hlL-STD- 1750,  any  operational  software  would  execute 
loperly  and,  hence,  revalidation  would  be  unnecessary.  However, 
ince  no  verification  method  is  of  perfect  quality,  revalidation  is 
necessary.  Any  architectural  discrepancies  which  are  discovered 
during  operational  software  revalidaticn  (which  remained  undetected  at 
the  time  of  verification)  require  corrections.  These  corrections 
extend  the  revalidation  process  and  increase  the  cost  of  validation. 
The  quality  of  the  verification  approach  will  determine  the  number  of 
architectural  discrepancies  which  remain  undetected  at  the  time  cl 
architectural  verif ica ticn,  which  in  turn,  determines  the  number  of 
architectural  discrepancies  which  will  be  discovered  during  software 
validation.  Therefore,  the  cost  of  revalidation  is  a  function  of  the 
quality  of  the  test  which  certifies  the  machine,  since  the  direct 
result  of  an  imperfect  verification  is  the  oversight  of  architectural 
discrepancies  which  must  be  fixed  and  which  extend  Mie  re velidaticn 
process.  (Presumably,  software  errors  have  been  discovered  in  the 
initial  validation  effort.) 

It  is  expected  that  these  costs  could  be  substantial.  If  an 
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architectural  discrepancy  is  discovered  in  the  flight  testing  phase  of 
software  validation,  a  significant  recovery  process  is  required. 
First,  sufficient  data  must  be  collected  tc  permit  identification  of 
tie  hardware  fault.  The  computer  must  be  physically  pulled  from  the 
airframe.  An  engineering  fix  fcr  the  problem  must  be  devised.  This 
fix  must  then  be  installed  in  the  machine.  The  machine  must  then  be 
retested  and  installed  in  the  airframe.  Additional  flight  tests  must 
be  performed  to  verify  that  the  engineering  change  both  solved  the 
problem  at  hand  and  did  not  create  any  new  problems.  The  software 
which  had  been  validated  to  date  must  be  re-checked  for  proper 
performance.  All  of  these  activities  can  add  up  tc  a  substantial 
penalty  in  time  and  money. 

The  ether  class  of  software  that  needs  tc  be  validated  is  new  software 
to  be  written.  Since  new  software  must  be  validated  in  any  case,  one 
might  argue  that  this  is  an  irrelevant  consideration.  However,  the 
cost  of  the  validation  will  certainly  be  dependent  on  the  quality  of 
the  test  method  used  to  certify  the  machine.  A  machine  that  has  been 
poorly  certified  may  have  many  architectural  discrepancies  which  must 
tse  found  and  corrected  in  the  course  of  operational  software 
validation  and,  conversely,  a  well  certified  machine  will  greatly 
reduce  the  exposure  to  unanticipated  software  validation  costs  because 
few  architectural  discrepancies  will  remain. 

New  software  validation  costs  may  be  divided  between  ccsts  incurred  in 
correcting  architectural  discrepancies  and  ccsts  incurred  in 
correcting  software  errors  (shown  in  the  right-haud  side  of  Figure 
f-2) .  The  new  software  validation  costs  of  interest  here,  i.e.,  those 
associated  with  the  quality  of  the  method  used  to  certify  the  machine, 
are  only  the  costs  of  finding  architectural  discrepancies  (software 
errors  have  nothing  to  do  with  the  quality  of  the  hardware 
verification).  Old  software  revalidation  costs  are  a  good 
approximation  for  these  new  software  ccsts  since  they  are  a  measure  of 
the  costs  associated  with  finding  architectural  discrepancies  only 
(see  the  left-hand  side  of  Figure  5-2)  . 


+ - + 


1  Existing  Software 

1  (to  be  captured) 

1 

1 

New  Software  1 

(to  be  written)  | 

1  None.  Software 

1 

No.  Costs  exist,  1 

1  previously 
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but  are  of  no  1 
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1  validated 
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1 
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1 

1 

1 

Figure  5-2.  Quality-Related  Cost  Components  of  Software  Validation 


5-5 


6176175R 


FINAL  FEPCET 


Fetruary  29,  1980 


In  summary,  the  implication  of  less  than  perfect  quality  of  a 
verification  method  is  that  additional  software  validation  costs  will 
be  incurred.  The  magnitude  of  these  costs  is  a  function  of  the  degree 
of  quality. 

Also  note  that  it  is  only  necessary  to  revalidate  operational 
software,  not  support  software.  A  Program  Manager  only  cares  that  the 
software  which  runs  on  his  machine  is  correct.  A  error  in  support 
software  which  affects  the  operational  software  would  be  caught  during 
revalidation.  Validating  the  operational  software  will,  in  effect, 
validate  the  support  software,  at  least  as  far  as  the  Program  Manager 
is  concerned.  Errors  in  support  software  which  do  not  affect  the 
operational  software  are  irrelevant. 


5.2. 1.1  Quality  Measurement 


In  order  to  project  software  validation  costs,  two  items  must  be 
estimated  -  the  relationship  between  quality  and  the  cost  of  software 
validation,  and  the  degree  of  quality  associated  with  each 
verification  method.  While  these  are  not  easy  tasks,  ignoring  quality 
and  software  validation  costs  (which  are  real  costs)  will  bias  the 
study  results  in  favor  of  the  cheaper,  less  complete  verification 
methods  and  would  result  in  ignoring  substantial  costs  to  the  Air 
Force.  The  following  method  is  used  to  estimate  quality. 

The  measure  or  the  quality  of  a  verification  method  is  defined  as  the 
r.umber  of  architectural  discrepancies  that  remain  in  a  machine  after 
trat  machine  has  been  certified  as  being  representative  of  the 
architecture.  That  is,  how  good  was  the  test  at  catching  all  of  the 
architectural  discrepancies  in  the  machine?  (Implicit  in  this  measure 
is  the  notion  that,  for  the  purpose  of  architecture  verification,  all 
machine  defects,  regardless  of  their  cause,  are  failures  to  meet  the 
architecture.  A  burned-out  gate  which  causes  a  multiply  failure  on  a 
particular  data  combination  is  as  much  a  failure  to  meet  the 
architecture  as  is  a  defective  algorithm  which  causes  a  multiply 
failure  on  a  particular  data  combinaticn. ) 

If  a  new  machine  is  designed  and  built  to  a  particular  architecture, 
and  then  tested  with  seme  architectural  verification  method,  many 
architectural  discrepancies  would  be  discovered  initially  in  the 
hardware  development  phase.  As  time  went  on,  the  number  of 
architectural  discrepancies  discovered  wculd  diminish.  Finally,  no 
more  architectural  discrepancies  would  be  discovered  and  the  machine 
wculd  be  deemed  architecturally  correct,  at  least  as  well  as  the 
verification  method  could  determine.  The  machine  would  then  be 
"sold-off".  However,  experience  would  suggest  that  some  hidden 
architectural  discrepancies  remain  in  the  machine.  Further  use  of  the 
machine,  such  as  in  software  validation  or  in  actual  flight  operation, 
wculd  uncover  some  of  these  hidden  architectural  discrepancies.  The 
rate  of  discovery  would  jump  after  "sell-off",  and  again,  the  rate  of 
discovery  would  be  a  decreasing  function  of  time  and  wculd  gradually 


5-6 


6'i76175A 


FINAL  EEPCFT 


Fctruary  29,  1980 


approach  zero.  This  error  discovery  scenario  is  depicted  in  Figure 
5-3.  The  number  of  architectural  discrepancies  remaining  after 
verification  (sell-off)  is  the  cross-hatched  area. 

For  machines  already  sold-off  and  in  use,  it  is  possible  to  estimate 
the  number  of  undiscovered  architectural  discrepancies  that  remain  at 
sell-off.  The  number  of  architectural  discrepancies  remaining  in  a 
machine  after  architectural  verification  is  the  sum  of  those 
architectural  discrepancies  discovered  to  date  (since  sell-off)  plus 
those  (as  yet)  undiscovered  architectural  discrepancies.  The  former 
are  recorded  in  the  Engineering  Changes  (FCs)  written  against  the 
machine.  The  latter  may  be  estimated  by  plotting  the  number  of  ECs 
with  time,  fitting  and  extrapolating  a  curve,  and  integrating  over 
time,  from  the  last  data  point  until  infinity.  This  is  shown 
graphically  in  Figure  5-4. 

The  number  of  architectural  discrepancies  found  will  te  a  function  of 
two  factors  -  complexity  of  the  machine  and  quality  of  test.  The 
crganizational/architectural  complexity  is  a  factor  relating  the 
number  of  remaining  errors  with  the  quality  of  the  verification 
method.  One  would  expect  that  the  number  of  errors  remaining  after 
certif icaticn  would  be  larger  for  a  larger  machine,  independent  of  the 
verification  method.  A  more  valid  quality  measure  is  therefore 
obtained  by  dividing  the  number  of  architectural  discrepancies 
remaining  in  the  machine  by  the  organizational  complexity,  which  can 
he  approximated  by  the  size  of  the  machine  (discussed  in  Appendix  B)  . 

The  resulting  quality  measure,  then,  of  any  particular  verification 
method  i,  as  measured  against  machine  j  is  expressed  as 


quality  =  f 
i 


ECs  since  sell-off  projected  remaining  discrepancies 

j  j 


machine  size 


Multiple  data  points  for  a  given  method  are  averaged  to  generate  a 
ccmpcsite  quality  estimate  for  that  method. 


5.2.  1.2  Engineering  Changes  and  Architectural  Discrepancies 


For  the  purposes  of  this  study,  the  concept  of  what  constitutes  an 
architectural  discrepancy  is  less  restrictive  than  the  concept 
normally  used  by  the  industry.  Any  machine  change  which,  if 
unprocessed,  would  cause  the  machine  to  fail  an  architecture 
verification  test  is  considered  to  be  an  architectural  discrepancy. 
This  includes  things  not  normally  associated  with  instruction  set 
architecture,  such  as  electrical  noise,  timing  races,  and  logic 
errors.  Also  included,  of  course,  are  more  obvious 
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Figure  5*-3.  Discovery  of  Architectural  Discrepancies  Over  Machine 

Lifetime 


Figure  5-4.  Hardware  Architectural  Discrepancies  Eenaining  After 

Architecture  Verification 
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architecture-related  problems  such  as  defective  algorithms  or 
improperly  implemented  microcode. 

Ike  reason  for  this  permissive  definition  is  twc-fold.  First, 
consider  the  certification  process  to  take  place  at  EEAfAC.  The  Air 
Force  cannot  discriminate  between  types  cf  failures  for  a  machine 
which  does  not  properly  execute  a  test.  Any  failure  must  be  a  failure 
to  meet  the  architecture,  regardless  of  the  cause.  The  second  reason 
is  that  all  of  these  failures  (including  those  not  nornally  associated 
with  instruction  set  architecture)  will  be  a  true  cost  to  the  Air 
Fcrce  during  Operational  Flight  Program  validation. 

This  is  not  to  suggest  that  all  Engineering  Changes  count  as 
architectural  discrepancies,  however.  ECs  ate  written  for  many 
reasons  including  changing  screws,  coatings  and  connectors,  as  well  as 
for  customer  directed  changes  and  cost  reductions.  These  types  of 
changes  are  not  relevant  to  this  study.  As  a  result,  each  EC  must  be 
scrutinized  as  to  its  relevance. 


5.2.  1.3  Estimating  the  Quality  -  Validation  Cost  Belationship 


As  discussed  earlier,  the  costs  cf  validating  operational  software  can 
he  broken  into  two  parts  -  the  cost  associated  with  correcting 
software  errors  and  the  cost  associated  with  correcting  hardware 
errors  (architectural  discrepancies) .  Only  the  latter  cost  is  of 
interest  in  this  study,  since  the  cost  associated  with  correcting 
software  errors  is  independent  of  the  quality  of  the  verification 
approach.  As  used  in  this  study,  the  term  ’’validation  cost"  means 
crly  those  validation  costs  associated  with  architectural 
discrepancies. 

The  expected  relationship  between  the  degree  of  quality  and  the  cost 
cf  validation  is  shown  in  Figure  5-5.  The  rationale  for  the  shape  of 
this  curve  is  as  follows.  A  near-perfect  test  would  incur  a  basic 
cost  for  validation  (e.g.,  flight  test),  but  should  proceed  smoothly 
with  few,  if  any,  defects  to  be  corrected.  Less  perfect  tests  would 
require  more  extensive  validation  efforts,  thereby  incurring  higher 
costs. 
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Perfect 

Quality 


Figure  5-5.  Validation  Costs  as  a  Function  of  Cuality 


lie  quality  -  software  validation  cost  relationship  can  be  estimated 
by  collecting  both  software  revalidation  cost  data  and  quality  data  on 
programs  that  have  incorporated  hardware  upgrades  to  existing  machines 
with  the  intention  of  capturing  existing  operational  software.  This 
yields  a  curve  which  has  only  those  costs  which  are  quality  related; 
the  software  validation  costs  associated  with  correcting  software 
errors  are  thereby  factored  out.  Since  the  existing  software  would 
have  already  been  validated,  the  cost  of  revalidation  could  be 
attributed  solely  to  those  architectural  discrepancies  which  remained 
in  the  machine  upon  completion  of  the  architectural  verification 
process.  That  is,  the  imperfect  quality  cf  the  verification  method 
was  the  only  reason  that  revalidation  costs  were  incurred.  (Note  that 
the  gathered  cost  data  must  be  scrutinized  to  see  that  no  costs  are 
included  for  the  addition  of  functional  capability.) 

It  is  not  necessary  to  obtain  quality  versus  software  validation  cost 
data  for  each  verification  method.  A  machine  that  has  been  verified 
to  a  certain  degree  (i.e. ,  according  to  the  quality  of  the  method 

used)  will  incur  a  software  validation  cost  commensurate  with  that 
quality,  regardless  of  the  verification  method  used.  An  architectural 
discrepancy  has  a  cost  of  correction  which  is  independent  of  the 
verification  method  used  on  that  machine.  Hence,  the  quality 

software  validation  cost  relationship,  regardless  of  which 
verification  method  was  used  to  estimate  that  relationship,  can  be 
applied  to  all  validation  methods. 
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The  software  validation  costs  are  a  function  of  the  aaount  of 
operational  software  to  be  validated.  fin  increase  in  the  number  of 
lines  of  code  would  result  in  a  proportional  increase  in  tne  cost  of 
validating  that  code.  Therefore,  the  software  validation  cost  aust  he 
normalized  by  the  number  of  the  lines  of  code. 


5.2. 1.3.1  Use  of  the  Quality  vs  Cost  Belation ship.  Application  of 
the  measured  quality  of  the  different  verification  approaches  to  the 
quality  versus  cost  relationship  yields  a  cost  (of  having 
less-than-perf ect,  quality)  for  each  of  the  verification  approaches. 

To  compute  these  costs,  EC  data  are  gathered  for  each  of  the 
verification  approaches.  Projections  are  then  made  to  estimate  the 
total  number  of  architectural  discrepancies  expected  to  remain  in  the 
machine  after  verification.  (These  two  steps  are  performed  in  the 
same  manner  as  is  used  to  estimate  the  quality  versus  cost 
relationship.)  These  projections  must  then  be  normalized  by  the  size 
of  the  machine  (discussed  in  a  later  section).  Finally,  each  of  these 
normalized  projections  is  applied  to  the  quality  versus  cost 
relationship  to  obtain  the  cost  for  each  of  the  verification 
approaches- 


5.2. 1.3.2  Summary  of  the  Quality  Concepts.  Quality  is  defined  to  be 
the  degree  to  which  an  architecture  verification  method  is  capable  of 
determining  compliance  of  a  hardware  implementation  to  an 
architectural  specification.  No  verification  method  is  of  perfect 
quality.  Less-than- perfect  quality  implies  that  architectural 
discrepancies  will  remain  in  a  machine  after  verification  testing  is 
scccessfully  completed  and  certification  is  granted.  These 
architectural  discrepancies  will  result  in  a  cost  to  the  Air  Force  in 
the  form  of  an  extended  operational  flight  software  validation 
process. 

Estimation  of  the  cost  penalty  due  to  less-than-perf ect  quality  for 
the  different  verification  methods  is  basically  a  two-step  process: 

Step  1.  Estimate  the  cost  versus  quality  relationship 
by 

(a)  gathering  EC  data  for  various  hardware 
upgrade  programs, 

(b)  gathering  software  validation  costs  for 
those  same  programs,  and 

(c)  fitting  curves  to  the  data  of  (a)  and  (b)  . 

Step  2.  Gather  EC  data  for  the  different  verification 

approaches  and  apply  to  the  cost  versus  quality 
relationship  obtained  in  Step  1. 
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5.2. 1.4  Objections  to  the  Quality/Sof t^iare  Validation  Cost  Approach 


It  may  be  argued  that  the  quality  data  vill  be  a  biased  measure  of  the 
true  quality  of  any  given  verification  method.  For  example,  consider 
the  Architecture  Verification  Program  (AVP)  method.  AVP  development 
is  typically  a  two  step  process.  The  first  step  is  to  write  the 
program  based  on  the  architecture  specification  document  (Principles 
of  Operation  hanual) .  This  will  test  a  certain  percentage  of  the 
machine  (i.e.,  it  will  have  a  certain  level  of  quality).  Since  it  is 
impossible  to  test  all  possible  data  combinations  and  since  many 
failures  are  data  related,  a  second  step  is  then  performed  and  the  AVP 
is  ined-tuned"  to  the  data  sensitivities  cf  the  particular  machine 
under  test.  The  machine  is  examined  to  verify  that  all  possible 
execution  paths  (which  are  data  dependent)  are  exercised  (i.e.,  there 
is  100  percent  microcode  coverage).  For  example,  a  multiply  algorithm 
would  he  examined  to  verify  that  the  different  test  cases  used  all 
possible  branch  paths  in  the  algorithm. 

This  fine-tuning  affords  an  increase  in  quality  for  most  approaches 
(except  Eandom)  which  is  unavailable  to  the  Air  Force  since  it  cannot 
be  determined  in  advance  what  algorithms  will  be  used  by  different 
manufacturers  to  implement  the  instruction  set.  This  results  in  an 
upward  bias  of  the  quality  estimate. 

While  this  is  a  valid  objection,  it  does  not  severely  limit  the  value 
cf  this  data,  however,  because  of  two  factors.  First,  it  is  probably 
a  uniform  bias.  That  is,  each  program  to  be  investigated  has  as  its 
gcal  a  high  degree  of  quality.  It  is  reasonable  to  expect  that  each 
program  used  this  relatively  inexpensive  method  (i.e.,  fine-tuning) 
for  improving  test  quality.  Second,  it  is  the  primary  goal  of  this 
study  to  select  the  best  method  relative  to  the  ethers. 

Another  objection  to  this  quality  -  software  validation  cost  approach 
is  that  a  large  number  cf  data  points  wculd  be  necessary  to  obtain  a 
high  degree  of  accuracy  in  cost  estimation.  This  is  also  a  valid 
objection.  However,  the  alternative  is  to  ignore  quality  and  software 
validation  costs;  but  experience  would  suggest  that  these  costs  can  be 
substantial.  It  is  better  to  make  use  cf  the  information  that  is 
available  (accepting  a  degree  of  uncertainty)  than  to  ignore  it. 


5.3  PEOJECIiCN  OF  THE  TOTAL  KUKEEF  CF  A FCEIT ECTUP AL  DISCREPANCIES 


Sppendix  B  describes  in  detail  the  method  for  projecting  the  total 
number  of  architectural  discrepancies  from  the  EC  data.  It  also 
describes  the  method  for  normalizing  the  data  by  machine  size. 
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6-0  VEPIFICAIION  APPROACHES 

A  number  of  ditfecent  afucoaches  have  beer,  identified  roE  utie  ir 
verifying  computer  architectures  and  have  teen  investigated; 

AN/AyK-15A  Acceptance  Test  Program  (ATE) 

Random  Instructions 
Analytical  Research 

Architectural  Verification  Program  (AVP)  -  System  3bO/37C 
Diagnostic 

functional  Test  Program  (FTP) 

Instruction  Set  Processor  (ISF) 

LocKStep 

Each  of  these  approaches  is  described  herein  as  it  currently  is  used, 
along  with  observations  regarding  its  applicability  tc  MIL-STD- 1 750. 
Fass/fail  criteria  are  also  applied  to  each  verification  approach.  A 
subseguent  section  describes  these  approaches  after  they  are  modified 
tc  apply  to  HIL-STD-1750.  Any  special  reauirements  for  impiemen tation 
are  also  described-  Finally,  the  methods  are  compared  and  the 
essential  differences  among  them  are  clarified. 


6.1  AN/AYK-15A  ACCEPTANCE  TEST  PSOGEAH 


6.1.1  Description 


The  AN/ATK-15A  Acceptance  Test  Plan  (ATP)  consists  cf  a  number  of 
tests  which  must  be  successfully  completed  in  order  to  sell-off  an 
AK/AYK-15A  computer.  These  tests  include  testing  the  MIL-STD-1750 
architecture  among  other  tests.  The  ATP  method  consists  of  extracting 
the  portion  of  the  ATP  which  tests  the  HIl-STD-1750  architecture  and 
using  this  as  the  aiL-STD-1750  architecture  verification  program. 

The  AN/AYK-15A  ATP  consists  of  a  number  of  subtests  which  must  be 
passed  as  part  of  the  sell-off  ptocedcre  for  the  AN/AYK-15A  computer. 
These  subtests  inc.lude:  User's  Console,  Instruction  Set,  Registers, 
Bain  Storage,  Bus  Controller,  Input/Output,  Power  ON/CFF  Sequencing, 
etc.  The  ATP  includes  some  subtests  which  are  not  part  of 
HIl-STD-1750.  These  subtests  (User’s  Console,  Eus  Controller,  Power 
CS/OFF  Sequencing)  would  be  eliminated  from  the  ATP  to  make  it  a 
vendor  independent  architecture  verification  program. 

he  Instruction  Set  test  verifies  OP  CODE  assignment  and  basic 

unctional  operation  of  each  instruction.  It  then  ohecks  that  the 
instruction  correctly  modifies,  or  does  not  alter,  as  the  case  may  be, 
the  status  word  for  all  possible  values  the  status  word  may  have. 
This  portion  of  the  test  is  exhaustive.  It  also  verifies  the  indexing 
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cacaLility  Lor  all  i  rist  ruct  i  cns  which  alloti  indexing.  It  checKs  the 
use  of  each  of  tne  15  index  registers  ard  uses  positive,  negative  ana 
2erc  values  lu  the  index  registers.  It  also  verifies  the  capah.iity 
cf  each  instruction  to  generate  an  urderfloe  and/oi  overflow 
interrupt,  when  required.  Data  values  used  in  the  Instruction  5et 
test  cover  ail  comDina ti cns  of  positive  and  negative  nuefeis. 

The  test  also  has  a  benchinark  portion  to  verify  instruction  tiBies. 
This  test  would  be  eliBi.".ated ,  since  no  tiling  is  specified  in 
PIL-STD- 1750.  There  is  also  a  randca  instructicn  hang  test  tc  verify 
the  handling  of  illegal  instructions. 

The  Pegister  test  verifies  the  capability  to  address,  set  and  reset 
various  registers  in  the  AN/AYK-15A.  Cr.ly  those  registers  defined  in 
KIL-TTD-  1750  would  be  included  in  this  irethod.  The  registers  checked 
include  the  CPU  general  registers,  Beiiory  Protect  BAH,  Status 
register,  and  Interrupt  Mask  register.  The  CPU  general  registers  and 
the  Status  registers  are  checked  by  running  all  6i4F  possible  data 
patterns  through  them. 

The  Hair  Storage  test  verifies  each  main  storage  location  by  writing 
ard  reading  each  of  the  64K  possible  data  patterns  in  eacn  location. 
The  test  also  verifies  the  main  storage  protection  features. 

The  Input/Output  tests  verify  the  operation  of  the  Timers,  Direct 
Input/cutput,  external  interrupts  and  DBA. 


6. 1.1.1  Block  Diagram 


Figure  6-1  depicts  a  hardware  block  diagram  for  this  approach. 
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F  .  1 .  2  Expec-iepce 

Ihe  ATP  appcodcn  tias  been  used  by  nary  lai;  u  t  ac  t  u  i  :  s  ,  ir.c^aa  n  4 
tc  verify  their  coaputers. 


€.1.3  Hardware  Beaources 

The  AK/AYK-15A  ATP  requires  the  conscle  (which  is  citioi.ai  i 
current  version  of  fl IL-STD- 17*^0)  and  special  hardwaLe  tu  test 
external  interrupts  and  CHA.  To  work  in  a  lultiiie  ve 
envircnaent,  the  hardware  interfaces  would  have  to  be  s tanoa i ai lec 


€.1.14  Software  Resources 

The  software  resources  for  the  AN/AYK-I'^A  already  exist,  aac 
effort  would  ne  to  select  only  those  pottiers  of  the  ATi  wiiich  ve 
the  architecture,  and  tc  add  tests  as  required  tc  incitaat 
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6.?  FANDCM  INSTfiUCTIONS 


6.2.1  Description 


This  is  a  statistical  approach  to  the  task  of  verification.  The 
candidate  computer  is  initialized  to  a  known  state.  A  block  of  random 
instructions  is  then  generated  and  executed  by  the  unit.  All  machine 
state  results  are  then  captured  and  ccmcared  against  the  expected 
results  which  ara  obtained  through  simulation.  This  process  is 
continued  until  atistical  evidence  of  completeness  is  acceptable. 

There  are  two  lain  programs  reauired  by  the  approach.  A  program  to 
generate  a  block  of  random  instructions  and  a  valid  simulator  to 
generate  the  expected  results.  Contrcl  for  the  twc  programs  is 
provided  in  a  table  of  implementaticn  dependent  conditions. 

A  random  seed  is  generated,  saved  for  future  recall  and  fed  to  the 
instruction  stream  generator  which  returns  a  variable  length  block  of 
random  instructions.  This  instruction  stream  is  fed  to  the  simulator 
along  with  the  control  table  of  inpleirentation  dependent  conditions 
and  the  initial  status  of  the  machine.  This  is  simulated  upon  the 
simulator,  and  the  ending  status  of  the  sequence  is  returned  for  use 
as  the  expected  values  for  the  computer  under  test.  The  instruction 
stream  is  then  run  on  the  hardware,  an  interrupt  is  forced  and  the 
status  variables  captured  for  comparison  with  the  expected  results. 
The  two  results  ate  compared  and,  if  equal,  the  next  test  is 
generated.  If  not  equal  the  stream  is  rerun  to  check  for  an 
intermittent  failure.  If  the  second  run  passes,  the  error  is  logged 
and  the  next  test  generated.  If  both  runs  fail,  the  last  instruction 
in  the  stream  is  replaced  by  a  NO-CP  and  the  stream  is  rerun  on  both 
the  hardware  and  on  the  simulator.  The  process  is  continued  up  the 
stream  until  a  successful  comparison  is  obtained.  The  last  NO-OP  is 
replaced  by  the  instruction  and  the  stream  run  on  the  hardware  and  on 
the  simulator  to  capture  expected  and  actual  results  at  each  stage  of 
the  stream.  This  information  is  then  printed  for  use  in  diagnosing 
the  failure. 


6.2. 1.1  clock  Diagram 

Figure  6-2  depicts  a  hardware  block  diagram  for  this  approach. 
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figure  6-2.  Randcm  Instructicrs  Block  Diagram 


6.2.2  Bxperieiice 


IBf!  has  used  this  method  to  check  the  completeness  cf  the  System 
36C/370  verification  method  and  found  that  detection  times  were 
relatively  short  when  known  problems  existed  in  the  unit.  Generally 
the  error  was  found  in  the  first  30  minutes  of  run  time  for  a  machine 
with  the  performance  in  the  range  of  four  millicn  instructions  per 
seccnd . 

Tycically  1,000,000  test  sets  of  randomly  generated  instructions  from 
1  to  32  instructions  in  length  (average  length  cf  10)  can  he  run  in  a 
6-hour  period.  Thus,  it  is  expected  that  10,000,000  instruction  cases 
are  executed  in  an  8-hcur  period  for  a  four  million  instructions  pti 
seccrl  computer. 

This  method  has  been  successfully  used  in  all  four  stages  or  computer 
development; 

1.  by  Engineering  as  a  bring  up  tocl 

2.  hy  Quality  Assurance  as  a  verification,  tool 
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3.  hy  Manolacturing  for  final  box  checkout 

4.  by  Field  Engineering  as  a  diagnostic  and  operational  check. 


€.2.3  Hardware  Besources 


7he  Pandcm  Instruction  method  requires  a  specialized  tester  to  verify 
T/C,  interrupt  structure  and  concurrent  operations.  This  tester  is 
designed  for  a  particular  computer  and  I/O  configuration.  Also,  the 
ccmDuter  must  have  a  special  interface  to  the  tester  so  that  the 
tester  can  be  controlled  by  the  test  program.  In  addition.  Ground 
Support  Equipment  is  necessary  for  program  ccntrcl  and  error 
reporting. 


6.2.4  Software  Besources 


A  complete  MIL-STD- 1 750  architectural  simulator  must  he  written  to 
generate  data  for  comparison.  Additional  advantages  are  gained  if  the 
simulator  is  written  to  run  on  the  RIl-STD-1750  computer  itself; 
however,  this  may  require  prevision  for  additional  main  storage.  For 
example,  I/O  transfers  are  substantially  reduced,  but  much  mote  main 
storage  is  required  for  the  simulator.  Some  type  of  an  I/O  tester  is 
also  needed  for  verification  of  I/O  related  architectural  features.  A 
tandem  instruction  generator  must  be  written  to  generate  the 
instructions  and  compare  the  results. 


€.2.5  Personnel 


A  person  who  is  experienced  in  architectural  verification  and  in 
simulation  is  required  to  write  the  pregram  for  the  MIL-STD- 1750 
simulator.  A  person  with  some  support  software  experience  and 
familiarity  with  the  MIL-STD-1750  architecture  is  required  to  write 
the  random  instruction  generator  program.  The  actual  execution  of  the 
test  pregram  can  be  performed  by  a  relatively  inexperienced  person. 
Error  analyses  could  require  a  person  with  a  high  degree  of  training 
in  architectural  verification,  although  error  analysis  is  probably  a 
contractor's  responsibility. 


€.2.6  Observations  and  Applicability  to  HII-STD- 1750 

The  Fandom  Instruction  method  provides  an  excellent  method  to  verify 
cemputer  architectures.  The  method  provides  for  a  thorough  testing  of 
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instruction  sequence  dependencies  and  other  subtleties  due  to  the 
random  nature  of  the  test  cases.  This  approach  also  shows  promise  as 
a  good  method  to  improve  confidence  and  completeness  it  other  methods. 

No  detailed  analysis  of  the  architecture  wculd  be  required  to  generate 
the  test  sequence. 

A  detailed  analytical  analysis  has  been  unsuccessful  at  developing  the 
criteria  used  for  completeness.  Engineering  judgement  has  determined 
the  number  of  instructions  necessary  for  the  verification  tests  to 
achieve  a  level  of  completeness;  the  randcm  process  can  be  terminated 
after  1,250,000  random  instructions. 

The  Fandom  Instruction  is  an  excellent  test  technique;  however,  tests 
cf  boundary  conditions  (like  a  branch  instruction  in  the  last  main 
storage  location)  would  have  to  be  added  to  completely  test  the 
KIL-STD-1750  architecture.  The  MIl-STD-1750  simulator  required  would 
have  to  completely  represent  the  architecture. 

The  Fandom  Instruction  method  does  not  contain  any  vendor  dependencies 
in  its  test  cases.  The  only  application  dependency  in  the  fiandom 
Instruction  method  is  that  the  main  storage  must  be  large  enough  to 
contain  the  program.  If  a  self-hosted  simulator  is  used  in  this 
method  the  main  storage  requirement  could  become  quite  large. 

Ihis  approach  passes  the  defined  pass/fail  criteria  and  will  be 
discussed  further  in  Section  7. 
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6.3  SNRLYTIC&L  APPROACH 


The  following  text  is  a  sumaacy  of  a  very  coapl^x  approach  and 
requires  a  background  in  this  field.  Further  inforaaticn  is  available 
from  appropriate  documents  located  via  the  Bibliography  (listed  under 
Carter) . 


6.3.1  Description 


This  method  consists  of  using  an  analytical  method  called  Symbolic 
Eiecution  to  verify  that  the  hardware  implementation  cf  the  computer 
meets  the  architectural  specification.  The  hardware  i nplementation  in 
the  form  of  microcode  or  finite  state  information  is  translated  into  a 
machine  readable  format  for  analysis  by  a  program  which  verifies  by 
Symbolic  Execution  that  it  conforms  to  the  architectural 
specification.  The  specification  must  also  be  supplied  in  some 
machine  readable  format. 

In  this  approach  two  definitions  are  required;  one  for  the 
architecture  specification  and  the  ether  for  the  machine 
implementation.  Both  must  be  given  in  the  architectural  description 
language  "Language  for  Symbolic  Simulation"  (LSS) .  These  definitions 
consist  of  a  facility  vector  describing  machine  cemponents  and  a 
.decision  tree  describing  the  control  structure.  The  facility  vector 
specifies  the  various  components  which  can  be  observed  or  manipulated 
by  the  machine.  The  operations  upon  the  facility  vector  components 
ate  specified  by  macro  routines  written  in  LSS  and  the  order  of  macro 
application  is  determined  by  the  control  tree  operation  algorithm. 
(That  is,  each  decision  tree  is  composed  cf  macros.)  For  example,  in 
the  following  control  tree  segment 


Hacrcs  n2,  h3  and  tl4  must  be  carried  cut  before  Ml  can  begin.  Macros 
82,  M3  and  M4  may  be  performed  in  any  order.  When  M2,  M3  and  M4  have 
been  performed.  Ml  is  initiated.  The  facility  vector  components  are 
treated  by  LSS  as  APL-like  variables  with  specified  dimensions. 

Because  the  two  facility  vectors  (specification  and  implementation) 
generally  have  different  components,  to  prove  correctness  it  is  first 
necessary  to  specify  a  relation  between  them.  One  machine  action 
simulates  another  if  everything  that  the  first  machine  can  do,  the 
second  can  do,  though  possibly  in  a  different  way.  The  states  of 
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correspondence  for  the  symbolic  simulaticn  are  determined  by  pairs  of 
tree  control  structures  called  simulation  control  points,  and 
predicates.  Predicates  describe  the  conditions  that  the  initial 
values  of  the  facility  vector  must  satisfy  in  order  to  be  in  a 
particular  state.  The  states  of  correspondence  specify  relations 
between  the  components  of  the  respective  facility  vectors  at  these 
pairs  of  points.  The  set  of  all  machine  states  pairs  and  their 
relation  forms  the  simulation  relation. 

Proofs  of  equivalence  generally  proceed  as  follows.  For  each  pair  of 
corresponding  machine  states  in  the  simulation  relation  we  must; 

1.  Assert  that  the  simulation  conditions  corresponding  to  each 
component  hold. 

2.  Run  each  abstract  machine,  following  all  paths  at  branch 
points  and  performing  symbolic  computation  until  another 
machine  state  is  reached. 

3.  Verify  that  the  new  pair  of  states  corresponds  to  a 
component  of  the  simulation  relation. 

h.  Prove  that  the  simulation  conditions  of  this  ccmponent  hold. 

In  such  an  execution  analysis  for  proving  equivalence,  facility  vector 
parameters  have  as  initial  values  either  symbolic  corstants  (sym.  -^Is 
representing  unknown  but  fixed  values) ,  cr  values  determined  from  the 
simulation  relation.  When  symbolic  constants  are  encountered  in 
expressions  being  computed  in  assignment  statements  as  part  of 
Symbolic  Execution,  the  value  assigned  is  an  expression  involvins> 
operators  and  symbolic  constants.  This  expression  is  always 
simplified  combining  like  parameters. 

When  symbolic  constants  occur  in  predicates  evaluated  to  determine 
possible  branches,  a  single  flow  of  control  may  not  be  able  to  be 
determined,  since  the  predicate  may  evaluate  to  a  Boolean  expression 
involving  symbolic  constants,  which  can  not  be  evaluated  to  "true"  or 
"false".  In  such  a  case  all  possible  logically  independent  results  of 
evaluating  the  predicate  must  be  considered.  The  program  doing  the 
symbolic  evaluation  will  now  generate  a  subgoal  for  each  independent 
result,  add  a  predicate  expressing  the  truth  of  the  result  to  the 
predicate  list,  and  simplify  the  result.  One  path  is  taken;  the 
remaining  paths  will  be  followed  later.  (If  the  predicate  involving 
symbolic  constants  evaluates  to  true  cr  false,  then  the  preceding 
multicath  analysis  is  not  required  in  that  only  the  one  path  analysis 
need  continue.) 

After  generating  a  series  of  sets  of  subgoals,  a  simulation  control 
pcint  for  this  level  is  reached.  Now  it  is  required  that  the  other 
irachine  is  run  until  a  simulation  control  point  for  that  level  is 
reached. 

At  this  point,  a  comparison  is  made  between  the  specification  result 
and  the  implementation  result.  That  is,  the  program  verifies  that  the 
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pair  of  control  points  reached  defines  a  component  of  the  simulation 
relation.  The  values  of  the  two  facility  vectors  are  substituted  into 
the  simulation  conditions  of  this  component  and,  using  the  predicate 
list,  all  of  these  conditions  are  proved  tc  hold. 

This  process  is  repeated  for  each  of  the  remaining  tranches  of  the 
tree.  In  the  process  a  complete  goal  tree  is  formed.  Hhen  a  complete 
goal  tree  is  formed,  the  analysis  has  been  successful. 

It  may  occur  that  a  correspondence  can  not  be  proved  and  thus,  the 
goal  tree  cannot  be  completed.  Then  an  error  must  be  sought  in  the 
code  or  in  one  of  the  descriptions.  In  addition  the  occurrence  of  an 
unexplained  branch  will  signal  the  presence  of  an  error. 


6  -  3. 1 . 1  Example 


The  following  example  is  provided  for  further  illustration  of  the 
concept  of  symbolic  execution; 

if  X<1  then  Y;  =  1-X  else  Y;  =  X-1 

Suppose  we  wish  to  prove  that  after  the  statement  is  executed 
Y  =  |X-1|.  The  object  of  the  verification  is  tc  construct  a  proof 

tree.  A  node  in  the  tree  represents  a  class  of  states  of  the  system 
at  one  point  in  time.  The  toot  describes  all  the  initial  states  in 
which  our  program  can  start  execution  --  one  state  for  each  pair  of 
initial  values  for  X  and  Y.  The  leaves  of  the  proof  tree  represent 
all  the  possible  final  states  —  for  each  leaf  we  have  to  prove  that 
the  states  represented  by  the  leaf  satisfy  Y  =  | X- 1 | .  A  branch  in  the 
proof  tree  represents  a  computation  path. 

In  our  example  the  tree  is  first  initialized  to  the  following  root; 

1.  facility  vector; 

X  contains  v 

Y  contains  w  (where  v  and  w  are  some  arbitrary  symbols) 

2.  Predicate;  empty 

3.  Control  point;  before  the  if  statement.  The  root  represents 
all  those  states  where  X  and  Y  contain  some  arbitrary  values 
V  and  w,  which  are  not  constrained  by  anything  (predicate 
list  is  empty) ,  and  where  the  if  statement  is  just  about  to 
be  executed. 

Symbolic  execution  is  used  to  construct  a  tree  of  nodes  representing 
states  reachable  from  the  root  by  executing  the  given  program.  The 
first  statement  is  an  if  statement.  Since  our  current  node  (the  root) 
represents  states  where  v< 1  as  well  as  states  where  v>1  we  must  split 
the  execution  into  two  cases  (i.e.  ,  two  sub-goals).  Tte  first  subgoal 
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represents  those  states  where  v<1.  It  has  the  same  state  vector  as 
tte  root,  its  predicate  is  v<l,  and  control  is  just  before  the 
assignment  Y  :=  1-X.  The  second  subgoal  represents  these  states  where 
v>1.  Again  it  has  the  same  state  vector  as  the  root,  its  predicate  is 
v>1,  and  control  is  just  before  the  assignment  Y  :=  X-1.  Now  we 

choose  one  of  these  new  leaves  as  the  current  node,  say,  the  first 
subgoal,  and  proceed.  The  assignment  Y  ;=  1-X  is  executed  by  changing 
the  contents  of  Y  to  the  expression  1-v  and  advancing  the  control 
pcirt.  Since  the  control  has  reached  the  end  of  the  program  we  are 
supposed  to  prove  the  assertion  Y  -  | X- 1 1 .  Or,  more  exactly,  it  must 

be  proved  that  the  assertion  holds  in  every  state  represented  by 

1.  Facility  vector: 

X  contains  v 
Y  contains  1-v 

2.  Predicate;  v<1 

3.  Control  point  at  the  end  of  program  (actually  irrelevant  for 
correctness) . 

Therefore  we  must  prove  that  the  predicate  v<1  implies  the  assertion 
with  X  and  Y  replaced  by  their  contents.  And  that  is  the  verification 
condition 

v<1  implies  1-v  *  |v-1|, 

which  is  clearly  true.  In  an  analogous  way  we  obtain  from  the  second 
case  the  verification  condition 

v>1  implies  v-1  =  |  v- 1 1  , 

which  is  also  true.  In  the  above  example  our  simulation  relation 
would  consist  of  two  components.  In  the  first  component  the  stopping 
point  identifies  the  beginning  just  before  the  if  statement;  the 
associated  assertion  is  true  (i.e.,  no  assumption  about  initial 
values).  In  the  second  component  the  stopping  point  identifies  the 
end  just  after  the  if  statement;  the  associated  assertion  is 
Y  =  |X-1|.  Thus,  the  goal  tree  is  completed. 
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t.3.1.2  Functional  Diagram 

A  functional  diagram  does  not  apply  for  this  approach. 


6.3.2  Experience 


7his  method  is  currently  being  investigated  by  Dr.  H.  Carter  at  IBa's 
Besearch  facility  in  Yorktown,  New  York. 


6.3.3  Hardware  Resources 


This  method  does  not  ceguire  any  specific  hardware  resources. 
(However,  it  does  presuppose  a  computer  is  available  with  support  of 
the  ISS  system.) 


6.3.4  Software  Resources 


Each  vendor's  implementation  as  well  as  the  architecture  specification 
wculd  have  to  be  described  in  a  machine  readable  format  such  as  LSS 
(language  for  Symbolic  Simulation).  Also  each  vendor's  implementation 
wculd  require  a  different  definition  of  the  simulation  relation. 


6.3.5  Personnel 


This  method  requires  personnel  who  are  highly  trained  in  the  area  of 
simulation,  and  ace  familiar  with  architecture  description  languages 
such  as  LSS. 


6.3.6  Observations  and  Applicability  to  HII-STD- 1750 


This  approach  shows  much  promise  for  providing  a  very  complete  design 
verification  of  the  architecture.  It  would  also  allow  designs  to  be 
verified  before  hardware  was  built.  Since  each  of  the  possible  paths 
it  the  proof  tree  are  followed  for  both  the  architecture  specification 
and  the  hardware  implementation,  the  method  will  generally  catch 
subtle  second  order  effects. 
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Ibis  could  be  a  very  high  cost  approach  since  each  vendor's 
i nplenentation  hould  have  to  be  described  in  a  machine  readanle 
format.  Also,  the  risk  in  using  this  arproach  is  very  high  because 
verification  using  this  approach  has  not  yet  been  demonstrated. 
Another  problem  is  that  this  method  verifies  that  the  abstract 
definition  of  the  hardware  conforms  to  the  formal  specification.  It 
does  not  validate  that  the  intended  functions  of  the  architecture  are 
built  into  the  hardware.  The  actual  hardware  would  still  need  to  be 
checked  for  wiring  errors,  noise,  timing  races,  etc.  Certification 
would  require  some  type  of  further  testing. 

Due  to  the  nature  of  the  tools  required,  this  method  is  judged  not 
feasible  at  this  time,  and  will  net  be  considered  for  further 
analysis. 
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e.U  AECHITECTUBAL  VERIFICATICN  PPCGEAM  -  SYSTEM  360/370 


6.4.1  Description 


Ihe  IBM  System/360  architecture  verif icaticr.  approach  consists  of  a 
set  cf  programs  uhich  allow  test  cases  to  te  written  in  a  procedural 
language.  These  test  cases  are  stored  on  magnetic  tape.  A 
supervisor,  which  is  resident  on  the  ccmputer  whcse  architecture  is  to 
te  validated,  reads  the  test  cases  from  the  tape,  executes  them,  and 
checks  for  correct  results.  Any  errors  found  are  printed  out. 

A  test  case  is  written  to  test  each  item  cf  the  architecture.  Each 
test  case  is  similar  to  what  an  engineer  would  normally  enter  by  hand 
tc  test  the  same  function.  Each  test  case  contains  the  setup  and 
expected  results  for  testing  one  characteristic  of  cne  instruction. 
In  some  cases,  primarily  I/O,  more  than  one  instruction  is  required 
fcr  proper  execution.  Each  test  case  is  written  in  such  a  way  as  to 
test  cnly  one  function  at  a  time.  The  supervisor  program  automates 
the  process  by  setting  up  the  system,  executing  the  test  case,  and 
comparing  the  expected  tc  the  actual  results. 

The  supervisor  is  a  4K  stand-alone  program  which  requires  a  printer, 
c:nsole,  and  some  type  of  mass  storage.  In  testing  I/O  channels, 
extensive  use  is  made  cf  a  program  controlled  I/O  Adapter  to  force 
parity  errors  and  specific  I/O  line  sequences.  This  supervisor  reads 
test  cases  from  tape,  and  then  sets  up  all  general  purpose,  floating 
pcint,  and  control  registers,  and  low  core  and  main  storage  as 
specified.  It  then  executes  the  test  case  instruction  and  initiates 
comparisons  with  the  expected  results.  Comparisons  include  any 
storage  areas  specified;  all  general  purpose,  control,  and  floating 
pcint  registers;  and  all  of  low  core  frcm  addresses  000-1FF.  Input 
test  cases  are  bypassed  if  they  require  features  that  are  not  on  the 
machine  being  tested.  Onless  otherwise  specified  in  the  test  case, 
all  instructions  are  repeated  via  the  IXECDTE  instruction  to  test  this 
feature  of  the  System  360/370  architecture. 

Fcr  successful  cases,  the  test  case  number  is  printed  out  for 
documentation  purposes.  When  an  error  is  detected,  the  supervisor 
prints  out  both  the  expected  and  the  actual  results  for  the  failing 
comparison.  Only  the  error  data  is  printed  and,  in  crder  to  analyze 
the  problem,  it  is  necessary  to  refer  to  the  test  case  listing. 

Tbe  verification  program  is  initiated  by  loading  the  supervisor 
program  frcm  a  card  reader  or  frcm  tape.  When  loading  is  complete, 
the  system  enters  a  wait  state.  The  user  console  is  used  to  start 
execution.  The  console  can  also  be  used  to  input  additional  test 
cases  when  desired. 
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€.4.1.1  Block  Diagram 

ligure  6-3  depicts  a  hardware  block  diagram  for  this  approach. 


Figure  6-3.  IBM  System/360  Rrchitectore  Verification  Flock  Diagram 


6.4.2  Experience 


This  method  is  used  by  IBM  to  validate  the  System/360  architecture  on 
all  IBM  360/370  computers.  This  method  is  extremely  successful  as 
demonstrated  by  success  of  the  System  360/370  architectural 
compatibility. 


€.4.3  Hardware  Resources 


This  method  requires  the  use  of  an  I/O  Adapter,  which  simulates  I/O 
channels,  to  permit  the  testing  of  the  360/370  I/O  channels  and  CPU 
instructions  which  affect  the  channels.  A  direct  I/C  wrap  cable  is 
also  required  to  verify  the  direct  control  feature.  A  console, 
printer  and  some  type  of  mass  storage  are  also  required  to  execute  the 
test  cases. 
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This  method  requires  a  control  program  (which  is  run  cn  tne  computer 
under  test),  the  test  cases,  and  support  programs  to  update  the  test 
cases . 


6 . 4 . E  Personnel 


A  person  who  is  experienced  at  architectural  verification  rs  required 
tc  write  the  test  cases.  The  actual  execution  of  the  test  cases  can 
te  performed  by  a  relatively  inexperienced  person.  A  person 
experienced  in  architectural  verification  could  be  reguirea  for  error 
analysis,  although  error  analysis  is  probably  a  contractor's 
responsibility. 


6.4.6  Observations  and  AcPlicability  tc  NIL-STE- 17  50 


This  approach  relies  on  a  large  number  of  test  cases  ard  experience  to 
provide  for  thorough  testing  of  the  architecture.  If  a  sufficient 
number  of  test  cases  ace  used,  it  can  provide  a  very  thorough  method 
to  verify  an  architecture.  The  evidence  of  its  success  is  the  high 
transportability  of  utilities  and  user  programs  among  members  of  the 
System  360/370  family.  Due  to  the  nature  of  the  test  cases  and  the 
structure  of  the  supervisor,  additional  tests  could  easily  be  added  to 
the  verification  program. 

This  method's  greatest  drawback  is  that  the  test  cases  must  be 
generated  manually.  The  appropriateness  of  the  test  cases  generated 
depends  on  the  insight  of  the  person  writing  them.  During  the  initial 
use  of  this  method,  many  features  of  the  architecture  may  remain 
unverified,  but  as  problems  are  found  and  test  cases  tc  uncover  these 
problems  are  added,  the  thoroughness  of  the  approach  increases.  After 
sufficient  time  has  passed,  this  approach  can  mature  tc  provide  a  very 
thorough  method  of  verifying  an  architecture. 

This  method  allows  extensive  testing  in  a  "quiet"  environment.  It 
does  not  provide  a  facility  to  test  interaction  between  channels  or 
between  channels  and  the  CPO.  This  method  tests  all  aspects  ox  each 
instruction  with  the  exception  of  data  dependency.  An  attempt  is 
made,  however,  to  use  data  that  either  has  caused  prcblems  in  other 
systems  or  is  suspected  of  being  critical.  Since  a  test  case  for 
every  bit  combination  for  each  instruction  would  te  astronomical,  any 
data  dependency  situations  that  are  suspected  are  coded  up  and  added 
as  they  occur. 

This  method  could  be  used  to  provide  a  vendor  and  application 
independent  verification  of  the  MIL-STD-1750  architecture.  The 
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DIAGNOSTIC 


6.5.1  Desctiption 


The  Diagnostic  approach  to  architecture  verification  is  to  make  use  of 
tie  iniividuai  manufacturer's  diagnostic  programs.  Ihe  verification 
program  is  createi  by  taking  the  diagnostic  programs  and  removing 
hardware  iipiementation  dependent  tests  ftcm  them  and  using  only  the 
temainir.q  parts  chat  are  functional  in  nature.  Because  of  this 
removal  of  nardware  dependent  tests,  some  additional  tests  may  need  to 
be  written  ir  crier  to  completely  verify  the  architecture. 

The  Diagnostic  method  consists  of  taking  each  manufacturer's 
diagnostic  program  for  his  computer  and  combining  them  into  one 
architecture  verification  pregram.  The  individual  diagnostics  cannot 
be  taken  in  total,  since  they  may  contain  implementation  dependent 
tests.  That  is,  various  manufacturers  may  have  microcoded  Built-In 
Test  Eouipment  (BITE)  tests  or  additional  hardware  features,  (not 
defined  in  dlL -STD-1750)  to  increase  the  diagnostic  capability  of  the 
computer.  Aitnouga  all  computers  are  KTl-STD- 1 750  compatible  from  a 
user's  viewpoint,  they  ate  not  identical  ftcm  a  diagnostic  viewpoint 
since  the  hardware  implementation  is  different.  Therefore,  the 
manufacturer's  diagnostics  must  have  all  non-MIL-STE- 1750  nardware 
implementation  dependent  differences  removed  from  them.  Cnee  this  is 
deno,  the  diagnostics  will  run  on  any  flIl-STC-1750  computer.  However, 
the  removal  of  the  implementation  dependent  tests  decreases  the 
coverage  of  the  resulting  Diagnostic  architecture  verification 
program.  Additional  tests  may  then  have  to  be  added  to  restore  the 
coverage  to  an  acceptable  level. 
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6-^. 1.1  Block  Diagram 


Figure  6-4  depicts  a  hardware  block  diagram  for  this  approacri. 


Forces 

Parity  Errors 
and  I/O  Test 
Sequences 


figure  6-4.  Diagncstic  Flock  Diagram 


6.5.2  Experience 


lie  architecture  of  the  Advanced  Signal  Frocessor  (ASP),  which  iias 
developed  by  the  IBM  Federal  Systems  Division  in  Manassas,  Virginia, 
was  verified  by  using  the  Diagnostic  approach.  This  method  was  also 
used  by  the  Army/Navy  Computer  Family  Architecture  project  in 
verifying  the  PDF-11/70  computer  architecture. 


6.5.3  Hardware  Besour ces 


Since  various  manufacturer's  have  added  microcode  and/or  additional 
hardware  tc  augment  the  diagncstic  capability  of  their  computer,  it 
would  be  userul  to  use  those  features  in  verifying  the  architecture. 
For  example,  if  a  manufacturer  implemented  a  means  of  setting  bits  in 
the  Interrupt  Pending  Register,  this  feature  could  be  used  to  assist 
in  verifying  the  interrupt  priority  structure.  However,  to  be  of  use 
in  architecture  verification,  the  feature  would  have  tc  be  implemented 
by  all  manufacturers.  This  means  that  the  feature  would  have  to 
tecome  part  of  MIL-STD- 1750 .  Therefore,  this  method  might  require 
extensions  to  MIL-STD-1750  rather  than  special  additicral  equipment. 
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6 . 5 . U  Software  Resources 


The  software  resources  required  by  this  method  are  the  diagnostic 
programs  designed  by  each  vendor.  These  programs  wculd  be  used  to 
produce  the  Diagnostic  verification  program. 


6.5.5  Personnel 


A  person  experienced  in  architectural  verification  with  some 
understanding  of  diagnostics  wculd  be  required  to  design  the  resulting 
program.  The  actual  execution  of  the  method  can  be  performed  by  a 
relatively  inexperienced  person.  A  person  experienced  in 
architectural  verification  could  he  required  for  error  analysis, 
although  error  analysis  is  probably  a  contractor's  responsibility. 


€.5.6  Cbservations  and  Applicability  to  mi-STE- 17  50 


A  disadvantage  with  this  method  is  that  it  relies  or  the  diagnostics 
to  be  written  in  a  functional,  rather  than  a  hardware-oriented  method. 
This  could  result  in  the  diagnostics  heccming  much  larger  and  more 
costly  than  might  be  justified.  Combining  functional  diagnostics  from 
several  vendors  may  or  may  not  provide  complete  verification  and  would 
certainly  provide  several  redundant  tests. 

The  Diagnostic  method  provides  a  good  source  of  data  to  test  the 
critical  points  of  algorithm  implementation  by  various  vendors. 
Howe/er,  significant  effort  would  be  involved  to  remove  implementation 
dependent  tests  from  each  vendor's  diagnostic  and  then  to  combine  the 
various  diagnostics  into  one  Diagnostic  verification  program.  Also, 
this  resulting  Diagnostic  program  wculd  have  to  be  evaluated  to 
determine  if  removing  the  i mplementa ticn  dependent  tests  seriously 
ccmcrcmised  its  usefulness.  If  this  was  found  tc  be  the  case, 
additional  effort  would  be  requited  to  formulate  additional  tests  to 
provide  better  coverage  of  the  architecture. 

The  Diagnostic  method  starts  out  inherently  vendor  dependent.  Each 
riagnostic  program  has  test  cases  which  make  use  of  implementation 
dependent  hardware  features  and  knowledge  of  the  implementation  to 
reduce  the  number  of  test  cases.  However,  these  dependencies  could  be 
removed  to  provide  the  final  Diagnostic  verification  program.  The 
cnly  application  dependency  in  the  resulting  Diagnostic  method  is  that 
the  memory  must  be  large  enough  to  contain  the  program. 
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If  extensions  were  added  to  KIL-STD-1750  to  include  EIIE  facilities, 
it  might  he  possible  tc  use  vendor's  diagnostics  without  extensive 
ircdif  ication. 

This  method  passes  the  defined  pass/fail  criteria  and  will  be  further 
discussed  in  Section  7. 
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6.6  FONCTIONAL  lEST  PF.OGPAM 


6.6.1  Description 


The  Functional  Test  Program  (FTP)  approach  verifies  the  architecture 
hy  performing  functional  level  testing  of  the  ccmplete  instruction 
repertoire,  the  main  storage,  the  I/O,  the  interrupt  structure  and  the 
concurrent  functions-  The  instruction  repertoire  is  verified  by 
exercising  all  instructions  and  their  formats  as  defined  in  the 
architectural  specification-  The  architectural  specification 
describes  each  function  to  the  level  of  detail  that  must  be  understood 
in  order  to  prepare  an  assembly  language  program  that  relies  on  that 
function.  Instructions  are  exercised  using  all  addressing  formats, 
registers,  interrupts,  and  condition  codes.  The  main  storage,  I/C, 
and  interrupts  are  also  tested  hy  exercising  their  functions  as 
defined  in  the  architectural  specification  manual.  Test  equipment  is 
required  for  the  man/machine  interface  (controlling,  loading)  and 
causing  interrupts,  I/O  and  exception  sequences. 

The  FTP  method  uses  a  center  cut  approach  to  irstruction  set 
verification.  First,  a  core  set  of  instructions  are  verified.  This 
core  set  is  then  used  to  verify  the  remaining  instructions.  Although 
all  addressing  formats,  registers,  interrupts  and  condition  codes  are 
exercised,  the  testing  is  not  exhaustive  for  all  data  patterns.  The 
necessity  of  exhaustive  testing  is  eliminated  by  knowledge  of  the 
hardware  implementation.  Also,  because  the  FTP  is  used  to  debug  the 
hardware  and  for  environmental  testing,  it  must  be  of  a  manageable 
size  and  cycle  in  a  short  period  of  time.  This,  therefore,  precludes 
the  use  of  exhaustive  testing. 

The  method  used  for  main  storage  testing  is  dependent  on  main  storage 
usage.  Normally,  the  read/write  portions  of  the  main  storage  test  do 
not  check  the  portions  of  the  main  storage  in  which  the  test  routine 
and  supervisor  reside.  These  parts  of  memory  are  checked  by 
checksums.  This  is  due  to  the  restriction  that  the  complete  FTP  be 
contained  in  the  main  storage  at  all  times. 

The  verification  of  I/O,  interrupt  structure  and  concurrent  function 
is  aided  by  the  use  of  specialized  testers.  The  tester  stimulates  the 
external  inputs  and  the  FTP  verifies  the  proper  operation  of  the 
computer.  For  external  outputs,  the  FTP  generates  the  outputs  and 
they  are  stored  in  the  tester  for  later  analysis  by  the  FTP.  This 
requires  that  there  be  an  I/O  test«E  interface  between  the  tester  and 
the  computer  under  test.  The  tester  is  also  able  to  generate  a  number 
of  error  conditions  so  that  proper  operation  of  the  computer  can  be 
verified. 

The  FTP  also  exercises  all  Built-In-Test  Fauipment  (BITE)  hardware  for 
proper  operation  and  to  detect  any  hardware  failures. 
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Ground  Support  Equipment  is  required  for  program  loading,  control  and 
error  reporting. 

6.6.  1.1  Block  Diagram 

Figure  6-5  depicts  a  hardware  block  diagram  for  this  approach. 


figure  6-5.  FTP  Block  Diagram 


6.6.2  Experience 


he  IBM  Federal  Systems  Division  has  made  extensive  use  of  this  method 
or  verifying  many  computer  architectures.  The  FTP  is  used  as  a 
development  tool  to  verify  that  the  hardware  implements  the 
architecture.  In  addition,  the  FTP  is  used  as  part  of  the  Customer 
Pcceptance  Test  Procedure,  as  a  reliability  demonstration  tool,  and  as 
an  engineering  evaluation  tool  during  hardware  bring-up. 


6.6.3  Hardware  Besources 


The  FTP  method  requires  a  specialized  tester  to  verify  I/O  operations, 
the  interrupt  structure  and  concurrent  operations.  This  tester  is 
designed  for  a  particular  computer  and  I/O  configuration.  Also,  the 
computer  must  have  a  special  interface  to  the  tester  sc  that  the  FTP 
can  control  the  tester.  In  addition.  Ground  Support  Equipment  is 
necessary  for  program  control  and  error  reporting. 
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6.6.4  Sof twace  Resources 


The  software  required  for  this  method  is  the  Functioral  Test  Frograin 
itself.  Since  the  test  cases  are  imbedded  in  the  program,  no  special 
facilities  are  required  to  add  or  update  test  cases.  However,  this 
makes  the  addition  or  updating  of  test  cases  more  time  consuming 
because  parts  of  the  FTP  would  have  to  be  reassembled  and  relinked. 


6.6.5  Personnel 


A  person  who  is  experienced  at  architectural  verification  is  required 
tc  design  the  FTP.  The  actual  executicr  of  the  FTP  can  be  performed 
by  a  relatively  inexperienced  person.  Error  analysis  could  require  a 
person  with  a  high  degree  of  training  in  architectural  verification, 
although  error  analysis  is  probably  a  contractor's  responsibility. 


6.6.6  Cbservations  and  Applicability  to  WII-STD-1750 


This  approach  has  been  used  by  the  lES  Federal  Systems  Division  to 
verify  its  military  computers.  If  the  implementation  of  the 
architecture  is  known,  then  use  of  the  FTP  method  for  verification 
yields  a  high  degree  of  confidence  at  a  reasonable  cost. 

In  lEM's  Federal  Systems  Division  experience,  abcut  90  percent 
(coverage)  of  the  hardware  will  be  exercised  if  only  the  architectural 
specification  is  used  to  develop  verification  tests.  The  reason  is 
that  certain  instructions  can  be  implemented  in  different  ways,  each 
cf  which  yields  correct  results  but  which  might  require  different 
types  of  tests  to  see  that  no  implementation  error  has  been  made.  The 
lEH  Federal  Systems  Division  has  found  that  FTPS  cover  tetter  than  99 
percent  of  the  hardware  after  utilizing  implementation  information. 
If  a  general  purpose  verification  tool  is  tc  be  developed  like  an  FTP, 
more  exhaustive  testing  will  be  required  given  that  the  methods  of 
implementation  cannot  be  predetermined. 
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Ibe  FTP  method  can  provide  a  very  cost  effective  method  to  verify  a 
ccmputer  architecture.  However,  to  be  effective  in  a  multiple  vendor 
environment,  tests  must  be  enhanced  to  make  the  testing  more 
exhaustive  since  the  hardware  implementation  is  no  longer  known.  An 
existing  FTP  may  contain  some  vendor  dependencies  in  its  test  cases. 
For  example,  the  use  of  a  Diagnostic  instruction  or  EITE  hardware 
facilities  would  make  an  FTP  vendor  dependent.  However,  these  test 
cases  could  easily  be  removed.  The  only  application  dependency  in  the 
FTP  method  is  that  the  memory  must  be  large  enough  to  contain  the 
pr cgra  m. 

This  method  passes  the  defined  pass/fail  criteria  and  will  be  given 
further  consideration  in  Section  7. 
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6.7  INSTBUCTION  SET  PPOCESSCF 


6.7.1  Description 


Tte  Instruction  Set  Processor  (ISP)  approach  consists  or  first 
specifying  the  architecture  of  a  machine  in  a  Computer  Hardware 
Description  Language  such  as  Instruction  Set  Processor  Specifications 
(ISPS) .  This  ISPS  description  must  include  all  of  the  information 
normally  found  in  the  architectural  specif icaticn  manual.  The  ISPS 
specification  is  then  used  as  input  to  a  program  which  automatically 
produces  test  cases  for  a  verification  program.  This  verification 
program  is  then  executed  on  the  computer  to  verify  conformity. 

To  date,  most  Computer  Hardware  Description  language  research  has  used 
the  ISP  family  of  languages  in  the  areas  of  description,  simulation, 
hardware  synthesis,  software  synthesis  and  verification.  Although 
other  Computer  Hardware  Description  Languages  could  be  used  for  this 
approach,  we  will  focus  on  the  latest  version  of  ISP  (ISFS). 

ISPS  was  developed  by  Carnegie  -  Mellcn  University  to  describe 
precisely  the  program  level  of  a  ccmputer.  The  ISPS  description 
provides  a  standard,  unambiguous  description  that  can  be  used  to 
specify  future  software/hardware  development  and  this  description  also 
provides  a  vehicle  for  defining  the  reauired  information  used  for 
testing  the  architecture.  An  ISPS  description  defines  the  storage 
elements  and  sequences  of  operations  of  a  processor.  An  ISPS 
description  consists  of  a  block  of  storage  declarations  and  sequences 
cf  register  or  control  transfer  operations. 

The  storage  elements  include  storage  available  to  the  programmer 
(e.g.,  general  registers  and  main  storage)  and  local  storage  for  the 
processor  (e.g.,  internal  registers  like  program  status).  The 
sequence  of  operations  includes  sequencing  control  of  the  data 
operations,  and  specifies  how  the  processor  fetches,  decodes,  and 
executes  instructions.  Storage  elements  are  organized  information 
structures;  for  example,  a  main  storage  might  consist  cf  a  number  of 
words,  each  of  a  number  of  characters,  each  of  a  number  of  bits,  or  a 
register  might  consist  simply  of  a  number  of  bits.  Storage  elements 
are  defined  in  ISPS  by  a  name  and  description  of  their  structure. 

Hain  storage  is  referenced,  other  than  in  storage  declarations,  by  its 
name,  qualified  by  an  index  if  it  is  a  multiword  main  storage. 
Multiword  main  storages  must  be  accessed  using  an  arithmetic 
expression.  Main  storage  references  can  be  further  qualified  with  a 
range  of  bits  to  be  selected. 

register  transfers  are  used  to  describe  the  data  operations  on  storage 
elements.  Since  most  operations  in  a  computer  result  in  modifications 
cf  bits  in  storage  elements,  each  action  in  a  data  transfer  sequence 
takes  the  form: 
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stora  ge-expreEsion< - da t a -expression 

The  data-express ion  describes  the  transf criaticn  of  information  and 
the  bit  pattern  tnat  is  to  be  placed  in  the  storage  element  designated 
ty  the  storage-expre ssicn.  Data  operators  in  ISP  include  transfer, 
relational,  arithmetic.  Boolean,  shift  and  concatenaticn  operators. 
All  data  operators  assume  unsigned  binary  representaticr . 

The  evoking  of  actions  can  be  contrclled  by  conditional  actions  of  the 
form: 

(IF  condition  =>  action-sequence) 

where  the  condition  (a  Boolean  expression  which  is  either  false  or 
true,  where  false  is  defined  as  0  and  true  as  any  ncn-zero  value) 
describes  when  the  action-sequence  will  be  evoked,  and  the 
action-sequence  describes  what  transformations  take  place  on  what 
elements. 

The  capability  to  select  one  of  a  list  of  alternative  statements  to  be 
executed  is  provided  by  the  DECODE  statement: 

DECODE  expression  =  statement- list 

where  the  value  of  the  expression  is  interpreted  as  ar  integer  and 
tsed  to  select  one  of  the  alternative  statements  in  the 
statement-list.  The  alternative  statements  are  net  censidered  to  be 
concurrent  activities,  but  are  a  list  of  statements  where  the  ith 
statement  is  executed  if  the  value  of  the  expression  is  equal  to  i. 

The  information  contained  in  the  ISP  description  is  used  as  input  to  a 
test  generator  program  which  determines,  from  the  storage  declarations 
and  register  control  transfer  operations,  what  tests  are  required  for 
each  instruction  or  operation,  and  then  generates  assembler  level  code 
covering  these  test  requirements. 

The  test  requirements  consists  of  checks  for  the  following: 

The  functional  results 

Values  in  registers  used  and  unused 

All  required  interrupt  mechanisms 

Kain  Storage  locaticns,  both  used  and  related 

Condition  codes,  set  or  unaffected 

Error  indicators 

Order  of  occurrence  of  predicted  events 

Possible  interference  due  tc  simultaneous  I/O  and/cr  interrupts. 

Sufficient  data  patterns  would  have  to  be  either  provided  to  the  test 
generator  as  input  or  be  created  by  the  generator. 
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The  generated  test  program  is  then  executed  to  verify  that  the 
ccmputer  meets  the  descr ipticn. 

Ground  Support  Equipment  is  necessary  for  pregram  loading,  control  and 
error  reporting. 


€.7.1.1  Block,  Diagram 

Figure  6-6  depicts  a  hardsare  diagram  for  this  approach. 


Figure  6-6.  ISP  Block  Diagram 


6.7.2  Experience 


ISPS  has  been  widely  investigated  by  the  research  community  as  a 
vehicle  for  computer  hardware  description.  IBM  is  currently 
developing  automatic  test  generation  methods  to  verify  architectures 
because  they  show  much  promise  for  future  use.  At  the  present  time, 
lEH  is  developing  PL/I  programs  which  generate  assembler  level  code  to 
test  limited  sets  of  instructions  for  the  BIL-STC-1750  and  NATO  ANACS 
architectures. 


6.7.3  Hardware  Besources 


The  ISP  method  requires  a  specialized  tester  to  verify  I/O  operations, 
the  interrupt  structure  and  concurrent  CPE  and  I/O  operations.  This 
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tester  is  designed  for  a  particular  ccmputer  and  I/C  configuration. 
Also,  the  computer  must  have  a  special  interface  to  the  tester  so  that 
tie  tester  can  be  controlled  by  the  Test  program.  In  addition.  Ground 
Scppcrt  Equipment  is  necessary  for  program  control  and  error 
reporting. 


6.7.4  Software  Resources 


A  complete  and  unambiguous  description  of  MIL-STE-1750  must  be  written 
in  ISP-  The  software  required  to  generate  the  Test  pregram  is  a  high 
order  language  program  capable  of  generating  all  necessary  tests  from 
key  words  and  symbols  in  the  ISP  description;  but  it  doesn't  exist  and 
needs  to  be  written. 


€-7.5  Personnel 


A  person  who  is  experienced  in  architectural  verification  and  compiler 
development  techniques  is  required  to  write  the  program  for  the  test 
generator.  The  actual  execution  of  the  test  cases  can  be  performed  by 
a  relatively  inexperienced  person.  Error  analysis  could  require  a 
person  with  a  high  degree  of  training  in  architectural  verification, 
although  error  analysis  is  probably  a  contractor's  responsibility. 


6.7.6  ebservations  and  Applicability  to  y.ll»STD-17E0 


This  approach  shows  much  premise  for  providing  a  very  complete 
verification  of  the  architecture,  but  still  has  the  "all  possible 
combinations"  limitation.  However,  at  this  time,  it  has  a  high  risk 
factor  associated  with  it,  because  all  of  the  tools  needed  for  its 
implementation  have  not  been  completely  develorf3.  The  benefit  of 
this  approacn  is  that  it  could  be  one  of  t  most  rigorous  and 
tborough  verification  methods. 

A  good  deal  ox  benefit  could  be  gained  by  using  a  Computer  Hardware 
lescription  Language  to  define  the  MI L-STE- '(7  50  architecture  even  if 
auto  test  generation  was  not  attempted. 

A  difficulty  with  the  ISP  method  is  that  Computer  Hardware  Description 
languages  are  still  research  tools-  The  success  of  the  method  will 
depend  on  the  maturity  and  quality  of  the  language  used.  ISPS  is 
known  to  have  some  inconsistencies  and  ambiguities  stemming  primarily 
from  the  lack  of  a  clear,  precise,  formal  semantic  definition  of  the 
language . 

Another  problem  is  that  automatic  test  generation  programs  are  also  in 
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th€  research  stages  and  have  net  yet  teen  used  to  generate  I/c  or 
irterrupt  tests- 

KIL-STD- 1 7 50  would  need  to  be  defined  in  terms  cf  ISP  and  the  test 
generation  program  developed  for  the  MIL-STD-1750  architecture. 

Ike  testing  techniques  used  in  the  ISP  approach  are  similar  to  those 
used  in  FTP  and  AVP  approaches.  The  main  difference  is  that  the 
architectural  analysis  effort  is  applied  in  writing  a  high  order 
language  program  which  can  determine,  from  the  architectural 
description  in  ISP,  all  the  tests  that  are  needed  tc  validate  the 
architecture.  This  approach  is  not  considered  tc  be  feasible  at  this 
time  since  the  tools  required  are  still  in  the  research  stage.  For 
this  reason  the  ISP  method  fails  the  pass/fail  criteria  and  it  will 
net  be  considered  for  further  analysis. 
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6.8  LOCKSIZP 


6.8.1  Description 


He  Lockstep  approach  consists  of  coirtaring  the  computer  under  test 
against  one  which  is  defined  as  conforming  to  the  architecture.  The 
procedure  consists  of  running  an  identical  program  on  computers  which 
are  synchronized  to  each  other.  Synchronization  can  he  at  any  timing 
level  or  machir  ’  state.  The  test  equipment  then  verifies  that  both 
commuters  produce  the  same  results  at  each  synchronizing  point.  Since 
one  of  the  computers  is  defined  as  meeting  the  architecture,  if  both 
produce  identical  results  when  running  the  same  program,  then  the 
second  computer  also  meets  the  architecture. 

For  a  complete  understanding  of  the  Lockstep  method,  the  facilities 
provided  by  the  Trace  interrupt  must  first  be  understood.  The  Trace 
interrupt,  if  enabled,  will  generate  an  interrupt  after  the  execution 
of  each,  instruction.  This  enables  the  software  to  Trace  the  execution 
of  the  program.  In  addition  to  generating  the  interrupt,  the  hardware 
also  stores  data  concerning  the  state  of  the  computer  into  fixed 
storage  locations.  The  data  saved  includes;  the  instruction  counter, 
storage  address  register,  condition  status,  and  the  contents  of  all 
ceneral  registers.  This  data  can  then  be  used  by  the  software  to 
tiace  the  execution  of  a  program.  The  data  stored  by  the  Trace 
interrupt  concerning  the  state  of  the  computer  is  the  most  importan  ■ 
part  of  the  Lockstep  method. 

The  Lockstep  method  consists  of  two  computers,  one  previously  verified 
ty  some  method,  which  is  called  the  "colden"  computer  and  runs  the 
tests,  and  the  other  computer  which  is  the  one  tc  be  verified.  The 
"golden"  computer  loads  the  test  to  be  run  from  the  diskette  and 
informs  the  computer  under  test  via  a  serial  channel  which  test  to 
lead  from  its  own  diskette.  The  test  is  run  on  both  computers  and  the 
Trace  results  are  saved  in  a  buffer  after  the  execution  of  each  of  the 
test  case  instructions.  Once  the  test  case  is  executed,  the  "golder" 
computer  then  reads  the  buffer  from  the  computer  under  test  via  the 
serial  channel.  It  then  compares  the  results  obteired  from  the 
computer  under  test  with  its  own  results  to  see  if  the  Trace  data  is 
identical.  Any  differences  in  the  Trace  data  indicate  an  error  and 
the  misccmparing  data  is  crinted  out. 

Both  computers  have  a  real-time  operating  system  running  in  them.. 
Executing  under  this  oper..ting  system  is  the  program  which  selects  the 
test  cases  to  be  run,  communicates  with  the  other  computer,  and 
compares  the  Trace  data. 
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6. a. 1.1  EIock  Diagram 


Figure  6-7  depicts  a  hardware  block  diagram  for  this  approach. 


Forces 
Parity  Errors 
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Compare  Program 

Control  Program 
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Control  Computer 
Under  Test 


Diskette 
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figure  6-7.  Lockstep  Flock  Diagram 


6.8.2  Experience 


Tbe  IBM  Corporation  has  used  this  method  to  verify  that  ail  Series/1 
computers  comply  to  the  Series/1  architecture.  It  was  also  used  by 


the  IBM  Federal  Systems  Division  on 
Shuttle  I/O  Processor. 


the  Verdin  Correlator 


3  Usee  by 
and  Space 
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6-8.3  Rardware  Resources 


This  nethcd  re.3uires  a  "golden”  computer  as  well  as  hardware  within 
the  computer  under  test  to  implement  the  Trace  facility.  This  Trace 
hardware  would  not  be  difficult  or  expensive  to  design  in  a  new 
computer,  but  it  could  very  difficult  to  add  on  to  an  existing 
computer.  In  addition,  seme  type  of  hardware  tester  is  needed  to 
check  I/O.  Also,  the  computer  must  have  facilities  for  attaching  a 
CFT  terminal,  some  type  of  mass  storage  (e.g.,  diskette),  and  a  serial 
channel. 


6 . 8 . U  Software  Resources 


This  method  requires  a  real-time  operating  system  to  perform  all 
communication  with  the  peripherals-  A  ccntrol  program,  and  the  test 
cases  to  be  executed  are  also  required. 


6.8.5  Personnel 


A  person  who  is  experienced  in  the  design  of  real-time  operatirg 
systems  is  required  to  design  the  operating  system  and  the  control 
program.  A  person  experienced  in  architectural  verification  is 
required  to  design  the  test  cases.  The  actual  execution  of  the  method 
can  be  performed  by  a  relatively  inexperienced  person.  A  person 
experienced  in  architectural  verificaticn  may  be  required  for  error 
analysis,  although  error  analysis  is  probably  a  contractor's 
respon  sibility . 


6.8.6  Observations  and  Applicability  to  EIl-STD- 1750 


The  hardware  Trace  facility  allows  the  comparisons  between  the  two 
computers  to  be  very  detailed  and  thcrough.  This  means  that  the 
verificaticn  procedure  will  have  a  high  degree  cf  confidence.  Alsc, 
cree  started,  the  procedure  is  completely  automated,  and  so  an 
operator  need  not  be  present  when  the  test  is  being  run. 

The  major  disadvantage  with  this  method  is  that  the  first  computer 
(golden)  must  be  verified  by  some  other  method.  Also,  the  test  cases 
most  be  generated  manually.  This  means  the  quality  cf  the  test  cases 
is  determined  by  the  person  developing  them.  If  the  Trace  interrupt 
facility  is  not  part  of  the  architecture,  the  addition  of  this 
facility  tc  the  architecture  could  be  a  major  problem. 

The  test  case  generation  and  execution  part  of  this  procedure  is 
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iiiilar  to  the  System/360  RVP  method.  The  main  difference  is  the 
EACH  interrupt  facility  and  the  use  of  a  second  computer  to  compare 
results.  It  is  possible  tc  modify  this  method  sc  that  only  the 
computer  under  test  is  needed.  Instead  of  comparing  data  to  that 
generated  by  the  "golden”  computer,  the  "golden"  computer's  data  can 
be  stored  on  an  external  device  (diskette)  and  the  computet  under  test 
can  compare  its  data  to  the  stored  data.  This  would  require  a  larger 
mass  storage  device.  The  Lockstep  method  does  not  contain  any  vendor 
dependencies  in  its  test  cases;  however,  unless  the  Trace  interrupt 
facility  is  standardized  in  the  architecture,  vender  dependencies 
could  arise  in  the  control  of  the  test  program.  The  only  application 
dependency  in  ' the  Lockstep  method  is  that  the  memory  must  be  large 
enough  to  contain  the  program. 

This  method  passes  the  defined  pass/fail  criteria  and  will  be  given 
ferther  consideration  in  Section  7. 
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6.S  SUMKABY 


The  eight  approaches  described  above  are  summarized  in  Table  6-1. 
They  can  be  grouped  intc  four  different  generic  types.  These  types 
are  Functional,  Random  Instruction,  lockstep,  and  Analytical.  The 
functional  type  consists  of  the  Functional  Test  Program,  Architectural 
Verification  Program,  Acceptance  Test  Program,  Diagnostic  Program,  and 
the  ISP  method. 

The  Functional  type  of  verification  procedure  consists  of  a  program 
Vih.ich  verifies  the  architecture  by  executing  a  number  of  test  cases 
which  test  the  architecture  at  a  functional  level.  The  origins  of  the 
program  vary  depending  on  the  method  used  to  generate  the  functional 
type  test.  The  Functional  Test  Program  (FTP)  method  starts  with  the 
FTP  which  is  used  to  debug  the  hardware  and  generalize  it.  The 
Architectural  Verification  Program  (AVP)  method  uses  the  AVP  which  is 
more  general  and  more  exhaustive  then  the  FTP.  The  Acceptance  Test 
Program  (ATP)  method  uses  the  ATP  which  is  usually  somewhere  in 
between  the  FTP  and  AVP  in  generality.  The  Diagnostic  Program  starts 
with  the  diagnostics  for  the  hardware  and  attempts  to  generalize  them 
and  remove  any  implementation  dependencies  from  the  test.  The  ISP 
method  attempts  to  generate  functional  tests  automatically  from  a 
detailed  description  of  the  architecture.  The  ISP  method  is  an 
automated  AVP. 

The  Random  Instruction  type  consists  of  automatically  generating 
random  sequences  of  instructions,  executing  them,  and  verifying  that 
the  proper  results  are  generated.  The  expected  results  are  determined 
by  simulating  the  random  sequence  of  instructions  on  a  simulator.  The 
main  effort  in  this  approach  is  to  design  a  simulator  which  models  the 
architecture  as  closely  as  possible.  The  architecture  must  be  modeled 
with  sufficient  accuracy  so  that  the  simulator  and  actual  hardware 
give  identical  results. 

The  Lockstep  type  consists  cf  running  a  functional  type  test  program 
cn  two  computers  and  comparing  the  results  of  the  test  run  on  the  two 
computers.  To  aid  in  the  testing,  a  hardware  trace  facility  is  added 
to  the  computers.  This  facility  allows  all  pertinent  data  concerning 
the  state  of  the  computer  to  be  saved  after  the  execution  of  each 
instruction.  This  state  information  is  saved  after  the  execution  of 
each  instruction  of  the  test  case  and  compared  between  the  two 
ccmputers.  Tiie  main  effort  in  this  approach  is  the  i  nplementatior.  of 
the  trace  hardware  and  designing  and  coding  the  functional  type  test 
program. 

The  Analytic  Research  type  consists  of  converting  the  architectural 
description  and  the  implementation  (microcode  and/or  Icgic  diagrams) 
into  symbolic  language  descriptions  and  symbolically  executing  both  to 
verify  that  they  produce  the  same  result.  This  method  relies  on 
complete  descriptions  of  both  the  architecture  and  the  implementation. 
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The  main  effort  in  this  approach  is  in  converting  tc  the  symbolic 
language  and  executing  it.  It  is  important  to  realize  that  this 
method  verifies  the  intended  implementation  (the  logic)  not  the  actual 
hardware.  Also,  this  method  could  be  used  for  verification  before  the 
actual  hardware  is  built. 


6 . S . 1  Pass/Fail  Evaluation 

The  approaches  were  evaluated  using  Vender  Independence,  Application 
Independence,  Feasibility,  Uniqueness  from  other  approaches. 
Availability  of  Information,  and  Testability  within  a  two  week  period 
tc  determine  viability  for  vise  as  a  verification  approach  with  the 
following  results. 

Two  approaches,  the  User  Oriented  Microprocessor  ATP  and  the  Adam 
ATP,  were  eliminated  due  tc  lack  of  information.  However, 
indications  are  that  they  are  similar  to  the  AN/AIK-15A  ATP. 

Although  the  Functional  approaches  yield  similar  verification 
approaches,  none  of  the  methods  were  judged  to  he  significantly 
similar  to  another  to  justify  failing  them  since  each  of  the 
functional  methods  meet  different  design  requirements. 

There  were  some  application  and  vendor  dependencies  in  some  of 
the  methods.  However,  these  dependencies  were  considered  to  be 
correctable. 

Two  approaches,  the  ISP  method  and  the  Analytic  method,  were 
eliminated  due  to  feasibility.  These  were  eliminated  necause  the 
tools  required  to  implement  the  method  were  not  sufficiently 
developed  for  use  at  this  time  or  in  the  near  future. 


Table  6-2  further  summarizes  the  pass/fail  analysis  presented  in  this 
section. 
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Table  6-2.  Pass/Fail  Analysis 


Method 

Pass/Fail 

Failure  Criteria 

Adam  ATP 

Fail 

Not 

Unique* 

AK/AYK-ISA  ATP 

Pass 

Pandcm  Instructions 

Pass 

Analytical  Approach 

Fail 

Not 

Feasible 

AVP 

Pass 

Diagnostic 

Pass 

FTP 

Pass 

ISP 

Fail 

Not 

Feasible 

Lockstep 

Pass 

Dser  Oriented 

Fail 

Not 

Unique* 

Micro  Computer  (ATP) 

♦  Subset  of  AN/AYK-15A  ATP 


Generic  Type 


Functional 
F  unctional 
Fandom 

Instruction 
Analytical 
F  unctional 
Functional 
Functional 
Functional 
Icckstep 
F  unctional 
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7.0  ANALISIS 


The  previous  section  provided  detailed  descriptions  of  the  different 
verification  approaches  and  applied  the  pass/fail  criteria  to  them. 
This  section  provides,  for  those  verification  approaches  that  remain, 
detailed  cost  information  and  the  quality  data,  as  required  by  the 
cost  model.  These  data  are  analyzed  and  the  results  interpreted. 

This  section  consists  of  four  parts.  The  first  discusses  the  various 
test  configurations.  The  second  presents  cost  data  for  the  different 
verification  approaches.  Next,  the  quality  data  is  presented  and 
analyzed.  Finally,  conclusions  are  drawn  based  on  the  first  three 
sections. 


7.1  TEST  CONFIGURATION 


Each  verification  approach  may  be  implemented  utilizing  a  variety  of 
test  configurations.  For  reasons  of  practicality  the  following 
generic  test  configurations  have  been  identified  as  being 
representative  of  ail  possible  test  configurations  in  degrees  of 
complexity : 

Manual  -  minimal  hardware  configuraticr 

Automatic  -  simple  I/O,  self-hosted  control 

Master/Slave  -  auxiliary  processor  to  support  testing. 

The  Manual  Test  Configuration  will  be  discussed  first,  followed  by  the 
Haster/Slave  Test  Configuration.  The  Automatic  Test  Configuration 
will  be  described  last,  with  arguments  presented  for  considering  it  as 
a  subset  of  the  Master/Slave  Test  Configuration.  Each  test 

configuration  is  further  broken  down  into  hardware  and  software 
components.  Vendor  supplied  and  Air  Force  supplied  items  are 
separately  identified.  Cost  items  are  also  identified  as  applying  to 
either  the  vendor  or  to  the  Air  Force. 


7.1.1  Manual  Test  Configuration 


7. 1.1.1  Description 


The  Manual  Test  Configuration  is,  by  definition,  the  minimal  hardware 
configuration  necessary  to  perform  adequate  self-documenting 
verification  to  the  MIL-STD-1750  architecture.  The  test  configuration 
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reauires  all  hardware  to  be  supplied  by  the  vendor,  and  makes  no 
equipment  requirements  of  the  Air  Force  other  than  the  power  to  run 
the  test  equipment  and  computer.  The  hardware  necessary  for  the 
manual  configuration  is  as  fellows: 

•  aiL-SID-1750  Computer 

•  Ground  Support  Equipment  including  the  following: 

-  Memory  Load  Facility 

Hardcopy  Device  (Printer  or  Typewriter  Terminal) 

Memory  and  Register  Display/Modify  Facility 

-  Program  Start  and  Stop/On  Compare  Facility 

Figure  7-1  depicts  a  typical  Manual  Testing  Configuration.  Software 
is  supplied  by  the  Air  Fcrce  and  the  vendor.  The  architecture 
verification  program  for  the  MIL-STD-1750  is  developed  by  the  Air 
Fcrce,  assuming  a  standard  subroutine  linkage  for  output  messages  to 
the  vendor  supplied  hardcopy  device.  The  vendor  must  develop  the 
output  subroutine. 


Figure  7-1.  Manual  Test  Configuration 
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He  certification  scenario  would  then  he 


as  follows: 


1.  The  Air  Force  gives  a  copy  of  the  verification  program 

source  code  to  the  vendor  and  an  output  sutroutine 

specification. 

2.  The  vendor  develops  the  sutroutine  to  handle  hardcopy 
output,  and  then  prepares  the  load  tapes  by  assembling 
source  and  link  editing  with  the  output  subroutine. 

3.  The  vendor  brings  the  computer,  test  equipment  and  progran 
load  tapes  (containing  load  mcdules  of  the  verification 
program)  to  SEAPAC. 

4.  Under  SEAFAC  observation  the  vendor  first  clears  main 
storage  to  all  zeros  (which  the  Air  Force  verifies) ,  and 
then  loads  and  executes  the  verification  program  which 
prints  out  the  results. 

5.  Various  random  memory  locations  are  inspected  to  verify  the 
integrity  of  the  verification  program  on  the  load  tape. 

6.  The  vendor  leaves  the  load  tapes  and  assemtler/link  editor 
listings  with  SEAFAC  for  archive  purposes. 


7. 1. 1. 2  Analysis 


The  Manual  Test  Configuration  offers  a  lew  cost  testing  environment 
for  the  Air  Force  to  conduct  the  MII-STD-1750  certification  process. 
The  hardcopy  device  provides  automatic  self-documettation  of  test 
cases  under  control  of  the  verification  program.  The  vendor  has 
pre-verification  offsite  testing  ability.  After  successfully  testing 
offsite,  onsite  compliance  to  the  verification  procedrre  should  be 
simpler.  The  risk  of  the  vendor  fraudulently  a.  difying  the 
verification  program  to  compensate  for  architectural  inadequacies  is 
offset  by  the  Air  Force’s  keeping  the  lead  tapes  for  future  reference 
if  necessary.  The  disadvantages  associated  with  this  test 
configuration  center  around  its  limitations  associated  with  manual 
irtervention.  If  the  verificaticn  program  requires  several  system 
leads,  then  the  memory  loading  procedure  could  beceme  a  critical 
factor.  If  several  leads  could  not  be  made  from  the  same  IPL  tape, 
several  tape  mounts  are  necessary.  This  data  will  be  described 
further  in  the  analysis  of  subsequent  proposed  verification 
approaches. 
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7. 1. 1. 3  Costs 


Air  Force  costs  associated  with  the  Manual  Testing  Ccnfiguration  are 
limited  to  software  development  and  naintenance  costs  and  normal 
certification  personnel  staffing  hours  as  required  ty  the  selected 
verification  program  and  testing  procedure.  Hardware  costs  to  the  Air 
Force  are  non-existent  since  all  hardware  is  supplied  hy  the  vendor- 

Vendor  costs  include  hardware  costs,  software  costs,  and  computer 
operator  cost.  The  hardware  necessary  for  the  manual  test 
configuration  can  be  considered  to  be  the  standard  equipment  used  for 
crmal  computer  development  with  the  exception  of  the  hardcopy  device, 
oftware  cost  would  include  the  development  of  the  output  subroutine 
and  the  generation  of  the  load  tapes.  Personnel  costs  would  simply 
cover  the  vendor  representative  presence  to  mount  the  load  tapes  and 
start  the  verification  program  running. 
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7.1.2  Baster/Slave  Test  Configuration 


7. 1.2.1  Description 


Tke  Master/Slave  Test  Configuration  is  the  most  ccnpler  hardware 
configuration  considered  tc  nerfoim  the  verification  to  the 
KIL-STD-1750  architecture.  The  test  configuration  requires  both 
vendor  and  Air  Force  supplied  hardware  and  software.  Ire  hardware 
associated  with  the  Master/Slave  Test  Configuration  is  as  follows; 


ilM 

Kaster  Computer  with  the 
following; 

-  Hardcopy  Device 

-  Auxiliary  Storage 

-  Console/Terminal 

-  Communications  link 

-  MIL-STD- 1 55 3 

-  RS-232 

NIL-STD-1750  Computer 

I/O  Channel  on  MIL-STD-nSO  or 
Channel  Adapter 
(i.e.,  1553,  ES232,  etc.) 

Ground  Support  Equipment  with 

-  Hain  Storage  Load  Facility 

-  Main  Storage  and  Register 

Display/Modify  Facility 

-  Program  Start  and  Stop 
on  Compare  Facility 


SUPPLIEF 
Air  Force 


V  endor 

V  endor 


Vendor 


The  software  necessary  to  carry  out  the  certification  under  the 
Kaster/Slave  Test  Configuration  would  he  as  fcllcws; 


Verification  Program  for 
KIL-SID-1750 

Bootstrap  Program  on 
HIL-STD-1750 

Control  Program  on  Master 
I/O  Interface  Test  Programs 
Utility  Programs 


Air  Force 

Air  Force 

Air  Force 
Air  Force 

(Standard  on  Master 
Computer) 
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Incut/Output  Subroutines  Vendor 

Ficure  7-2  depicts  the  Kaster/Slave  Test  Configuration.  The 
certification  scenario  would  be  as  follows: 

1.  The  Air  Force  gives  a  copy  of  the  bootstrap  prograis  source 
code  and  I/O  Test  Programs  (both  require  vendor  supplied  I/O 
subroutines)  to  the  vendor  and  specifies  1/0  subroutine 
requirements. 

2.  The  vendor  develops  subroutines  to  handle  I/C,  and  then 
prepares  a  bootstrap  load  tape  by  assembling  bootstrap 
program  source  and  link  editing  it  with  the  I/C  subroutines. 
An  I/O  Test  Program  load  tape  is  similarly  developed. 

3.  The  vendor  brings  computer  test  equipment  and  load  tapes  to 
SIAFAC. 

U.  Both  the  vendor  and  the  Air  Force  connect  the  Master 

computer  to  the  vendor’s  I/O  channel. 

f.  The  I/O  Interface  Test  Programs  are  loaded  and  run  to  verify 
the  communication  interface  between  the  unit  under  test  and 
the  Master  computer. 

6.  Under  SEAFAC  observation  the  vendor  loads  the  bootstrap  and 
I/O  programs  and  starts  bcctstrap  execution. 

7.  SEAFAC  personnel  start  the  control  program  on  master 
computer. 

The  verification  program  is  transmitted  to  the  vendor’s 
MIL-STD-1750  computer  by  the  Master  computer  and  control  is 
given  to  the  verification  program.  When  the  verification 
program  finds  an  error  or  reauests  more  test  cases,  the 
Master  computer  is  notified  over  the  ccmmur.ication  link. 
Test  results  are  documented  or  seme  form  of  hardcopy  device 
(typewriter  terminal  or  printer)  assoc..ated  with  the  Master 
computer  system. 


7 , 1 . 2 . 2  Analysis 


The  Haster/Slave  Test  Configuration  offers  the  greatest  degree  of 
automation  available  for  the  Air  Force  to  conduct  the  MIL-STD-1750 
certification  process.  Furthermore,  this  approach  makes  certain 
verification  approaches  feasible  (Fandom,  AVP,  Lockstep)  that  would 
not  have  been  feasible  due  to  iirplementation  restrictions  discussed  in 
suhseauent  sections  of  this  document.  The  Master/Slave  Test 
Configuration  also  makes  use  of  the  available  peripherals  associated 
with  the  Master  computet  and  facilitates  excellent  tracking  capability 
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(ex.  spooling  intermediate  results  to  mars  storage  to  te  printed  at  a 
later  time).  The  Master/Slave  Test  Configuration  trines  into  play,  as 
'■ell,  the  powerful  computation  power  cf  the  Master  and  associated 
support  software. 

The  disadvantages  associated  with  this  test  configuration  are  cost  and 
complexity.  The  cost  of  the  Master  cemputer  is  considerable  under 
rest  circumstances,  but  for  purposes  of  this  study,  the  cost  of  the 
Master  computer,  its  peripherals  and  the  MIL-STD-1553  or  RS-232 
communication  link  are  to  be  considered  as  zero  since  they  are 
existing  SEAFAC  assets.  From  the  vendor's  perspective,  the  in, pact  of 
the  Master/Slave  Test  Configuration  (besides  the  additional 
input/output  routines  and  1/0  channel  reauired)  is  the  potential 
limitaticn  concerning  pretest.  Depending  on  the  verification  approach 
selected  to  run  under  this  test  conf iguraticn,  the  vender  will  be  able 
tc  utilize  parts  of  the  test  code  made  available  prior  to  in-house 
testing  by  the  Air  Force  at  SEAFAC.  In  the  area  of  complexity,  the 
Master/Slave  Test  Configuration  puts  an  additional  cost  burden  on  the 
Air  Force  for  developing  the  bootstrap  program  for  the  MIL-STD-1750 
and  control  program  for  the  Master  computer,  as  well  as  the  I/O 
Interface  Test  Programs. 


7 . 1 . 2 . 3  Costs 


Air  Force's  cost  associated  with  the  Master/Slave  Test  Configuration 
can  te  segmented  :,.nto  two  areas;  (1)  hardware  costs  and  (2)  software 
costs.  The  nardwere  costs  to  the  Air  Force  would  include  the  cost  of 
the  Master  computer  with  related  peripherals  and  com n unications  link 
which,  have  already  been  established  tc  be  zero.  The  software  costs 
are  compounded  by  the  additional  development  costs  of  the  bootstrap 
program  for  the  unit  under  test  and  the  control  program  for  the  Master 
cemputer  along  with  the  normal  cost  of  writing  (and  maintaining)  the 
selected  verification  program.  The  I/O  Interface  test  programs  are 
necessary  to  integrate  the  vendor  and  the  Air  Force  hardware  prior  to 
actual  verificatic  . 

SEAFAC  personnel  requirements  tc  monitor  the  execution  of  the 
verification  program  would  be  minimal  since  the  approach  is  fully- 
automated.  A  technician  familiar  with  the  Air  Force's  I/C  Interface 
shculd  he  available  to  assist  the  vendor  when  the  vender  connects  to 
that  I/O  Interface.  Vender  costs  consists  of  the  normal  comiuter  and 
Ground  Support  Equipment  augmented  by  the  I/O  channel  requirement. 
Software  provided  by  the  vendor  consist  cf  the  I/O  subroutines  and  the 
bcctstrap  and  I/O  Test  Pregram  load  tapes. 
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7.1.3  Automatic  j.ee^  .  .'onf iguration 


7. 1.3.1  OescriptiOD 


The  Automatic  Test  Configuration  provides  a  workable  standalone 
self-documenting  system,  with  reasonable  cost  and  performance  to 
conduct  the  certification  of  the  MIl-STD-1750  architecture.  This  test 
configuration  places  the  sane  software  and  hardware  reguirements  on 
the  vendor  as  described  in  the  Saster/Slave  Test  Configuration.  The 
hardware  associated  with  the  automatic  test  configuration  is  as 
f  ollciis; 


ITEH 

Certification  Peripherals 

-  Hardcopy  Device 

-  Auxiliary  Storage  (Tape/Floppy  Disk) 

-  I/O  Adapter  (MIL-STE-155 3/BS-232) 


Air  Force 
Air  Force 
Air  Force 
Air  force 


MIL-STD-1750  Computer 


Vendor 


I/O  Channel  or  I/O  Adapter  on  HIL-STD-1750  Vendor 

(HIL-SID-1553/RS-232) 

Ground  Support  Equipment  Vendor 

-  Main  Storage  Load  Facility 

-  Main  Storage  and  Register 

Display/Modify  Facility 

-  Program  Start  and  Stop  on  Compare  Facility 


The  software  necessary  to  carry  out  the  verification  under  the 
Automatic  Test  Configuration  would  be  as  fellows: 
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Verification  Program  for  IlIL-S'ID-1750 
Bootstrap  Program  on  MIL-STE-1750 
Contrcl  Program  to  Access  Auxiliary  Storage 
I/O  Interface  Test  Programs 
Input/output  Subroutines 
Utility  Programs 


Figure  7-3  depicts  the  Automatic  Test  Configuration. 


Air  force 

Air  Force 

Air  Force 

Air  Force 

Vendor 

(Standard  on 
Development 
Computer) 


Figure  7-3.  Automatic  Test  Configuration 


^te  certification  scenario  would  be  as  fellows: 

1.  The  Air  Force  gives  a  copy  of  the  bootstrap  program  source 
code  and  I/O  Test  Program  (both  require  vendor  supplied  I/C 
subroutines)  to  the  vendor  and  specifies  I/O  subroutine 
requirements. 

2.  The  vendor  developes  subroutines  to  handle  I/O,  and  then 
prepares  a  bootstrap  load  tape  by  assemhlinc  the  bootstrap 
program  source  and  link  editing  it  with  I/O  subroutines.  An 


7-10 


en6n5\ 


FINAL  BEPCBT 


February  29,  1980 


I/O  Test  Program  load  tape  is  similarly  developed. 

3.  The  vendor  brings  his  computer,  test  eguipmect  and  load  tape 
with  the  bootstrap  program  and  I/O  subroutines  to  SEAFAC. 

4.  Both  the  vendor  and  the  Air  Force  connect  SEAFAC's 
certification  peripherals  to  the  vendor's  I/C  channel. 

5.  I/O  Interface  test  programs  are  loaded  and  run  to  verify  the 
communication  interface  between  the  unit  under  test  and  the 
Air  force  supplied  peripherals. 

6.  Under  SEAFAC  observation,  the  vendor  loads  the  bootstrap  and 
I/O  programs,  and  starts  their  execution. 

1.  The  bootstrap  program  loads  the  verification  program  from 
the  auxiliary  storage  and  commences  execution.  Errors  or 
other  messages  will  be  logged  out  on  the  printer. 
Subsequent  program  loads  from  auxiliary  storage  will  be  made 
under  program  control.  No  manual  intervention  is  necessary. 


7. 1.3. 2  Analysis 


The  Automatic  Test  Configuration  offers  the  simplest  fully  automated, 
self-documenting  system  for  the  Air  Force  to  conduct  the  aiL-STD-1750 
certification  process.  This  approach  facilitates  certain  verification 
approaches,  but  limits  the  potential  dynamic  nature  of  certain  test 
procedures  (fiandom)  which  will  be  explained  in  subsequent  sections. 
The  main  disadvantage  of  this  approach  is  that  it  requires  the  Air 
Force  to  purchase,  integrate  and  maintain  the  selected  peripheral 
devices.  Vendor  requirements  are  the  sane  as  for  the  haster/Slave 
Test  Configuration,  but  the  implications  of  pre- verification  occurring 
offsite  using  the  Air  Force  developed  verification  program  becomes 
more  feasible  since  the  Automatic  Test  Configuration  is  easier  for  the 
vendor  to  implement  than  the  Haster/Slave  Test  configuration.  The 
Automatic  Test  Configuration  might  also  be  considered  portable  if  the 
Air  Force  implemented  it  as  such. 

In  summary  the  Automatic  Test  Configuration  will  give  the  Air  Force 
considerably  less  function  than  the  Haster/Slave  Test  Configuration, 
at  the  same  time  costing  more  because  of  the  purchase  of  additional 
peripheral  devices.  The  vendor  requirements  are  the  sane  and  the  Air 
Force's  software  costs  are  the  same  as  for  the  Haster/Slave  Test 
Configuration,  thus  indicating  that  the  Haster/Slave  Test 
Configuration  offers  a  superior  type  of  approach. 
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7. 1.3. 3  Costs 


The  cost  to  the  Air  Force  tc  inplenient  the  Automatic  Test 
Configuration  to  support  the  certification  of  the  MIL-STD- 1750 
computer  consists  of  the  following  items: 

•  Purchase  of  Certification  Peripherals  (Auxiliary  Storage  and 
Printer) 

•  Development  of  Interface 

•  Development  of  Verification  Program 

•  Development  of  Bootstrap  Program 

•  Development  of  the  I/O  Interface  Test  Programs 

•  haintenance  of  all  Software 

Vendor  costs  would  be  the  same  as  with  the  Haster/Slave  approach. 
SIAFAC  personnel  requirements,  to  monitor  the  execution  or  the 
certification  program,  would  be  minimal  since  the  approach  is  fully 
automated.  A  dynamic  approach  could  be  selected  requiring  the 
substitution  of  auxiliary  storage  to  handle  additional  data 
requirements  (floppy  disks  or  tapes),  thus  reducing  the  system  to  a 
semi- automated  state.  The  I/O  Interface  bring-up  would  require  a 
technician  familiar  with  the  Air  Force's  I/C  Interface  to  assist  the 
vendor. 


7 . 1 . U  Comparison 


All  three  approaches  described  in  the  preceding  sections  provide  the 
Air  Force  with  some  form  of  certification  facility.  The  hanual  Test 
Configuration  offers  the  least  amount  of  automaticn  while  the 
Master/Slave  Test  Configuration  has  the  most.  The  Manual  Test 
Cenf iguration  restricts  the  inplementaticn  cf  the  verification  program 
to  certain  verification  approaches,  while  the  Master/Slave  Test 
Configuration  places  no  limitations  on  the  verification  approach 
selected.  The  Manual  Test  Conf igurat icn  provides  convenient  offsite 
pre-testing  by  the  vender,  while  the  Master/Slave  Test  Configuration 
has  some  potential  implementaticn  dependent  limitations. 

The  Automatic  Test  Configuration  costs  more  for  the  Air  Force  to 
implement  because  of  the  additional  hardware  purchased,  while 
providing  less  function  than  the  Master/Slave  Test  Configuration.  For 
this  reason,  the  Automatic  Test  Configuration  will  be  dropped  from 
further  consideration  as  a  feasible  test  configuration.  Subsequent 
verification  approach  analysis  will  center  around  the  remaining  two. 
Table  7> 1  summarizes  the  cost  breakdewrs  and  components  associated 
with  each  test  configuration. 
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Table  7-1.  Test  Eleirents 


I  I  Haster/Slave  |  Autcsatic 


Vendor  Costs 
Software 


Persornel 

Hardware 


1 

I 

I  I/O  Subroutines 
(Load  Tape 
I  Support  Software 
I 

JGSE  Operator 
J 

1 1750  Computer 
I  Special  I/O 
I  Interface 
1 

Idaster  Computer 
I  (For  Verifica- 
I  tion  Pre-Test) 


I/O  Subroutines 
Load  Tape 
Support  Software 

GSE  Operator 

1750  Computer 
Special  I/O 
Interface 

Peripherals 

(For  Verifica¬ 
tion  Pre-Test) 


I  banual 


Output  Subroutine 
Lead  Tape 
Support  Software 

GSE  Operator 

1750  Computer 
None 


Printer 


3 
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Table  7-1.  Test  Elenents  (coot) 


1 

1  Master/Slave 

(  Automatic 

(  Manual 

1 Jir  force  Costs 

1 

1 

t 

1  Software 

1 

j  Verification 

1 

1  Verification 

( 

(Verification 

1 

1 

1  Program 

1 

1  Program 

1 

(  Program 

1 

1 

^Bootstrap  Source 

1 

(Bootstrap  Source 

( None 

1 

jControl  Program 

(Utility  Program  - 

(Utility  Program  - 

1 

i 

(Offload  Test  Code 

(  Offload  Source 

1 

jsupport  Software 

(Support  Software 

(Support  Software 

1 

iI/0  Test 

(I/O  Test 

I 

(I/O  Test 

1 

[ 

1  Hardware 

1 

llnterface 

1 

(Interface 

1 

1  Kcne 

! 

1 

1 

1  Master  Computer/ 

1 

( Peripherals 

1 

(Bene 

1 

1 

1  Peripherals 

1 

1 

1 

1 

1 

1 

1  Personnel 

tDevelcpment 

1 

(Development 

1 

( Development 

1 

1 

1  PrograiDDers 

1 

(  Programmers 

1 

(  Programmers 

J 

1 

1 

1 

1  Maintenance 

1 

( Maintenance 

1 

(Maintenance 

! 

1 

i  Programmers 

1 

(  Programmers 

1 

{  Programmers 

1 

1 

1 

1 

(lest  Operator/ 

1 

ITest  Cperator/ 

1 

(Test  Operator/ 

1 

1  Obser'ver 

(  Observer 

(  Observer 

1 

1 

; 

1 

(Integration 

1 

lintegration 

1 

(Bene 

! 

1  Technician 

(  Teohnician 

1 

1 

1  other 

1 

(Test  Procedure 

1 

(Test  Procedure 

1 

(Test  Procedure 

1 

1 

i 

(Utilize  Master 

1 

(Generate  Test 

1  Ncne 

1 

(  Computer  for 

1  Tape 

1 

1 

(  Test 

1 

1 
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7.2  TEST  APPHOACHES 


In  this  section,  the  test  appcoaches  renaining  after  applying  the 
pass/fail  criteria  will  be  discussed  under  tvo  test  configurations, 
Hanual  and  aastec/Slave.  Each  approach  vill  be  analyzed  and  described 
in  perspective  with  “being  irpleiented"  as  a  verification  program  for 
the  HIL-STD-1750.  Variations  of  certain  approaches  will  also  be 
covered.  The  proposed  approaches  discussed  in  the  following  sections 
are: 

•  AN/AYK-15A  Acceptance  Test  Program 

•  Bandom  Instructions 

•  Architectural  Verification  Program  (AVP)  -  System  360/370 

•  Diagnostic 

•  Functional  Test  Program  (FTP) 

with  existing  FTP  available 
with  no  FTP  available 

•  Lockstep 

-  with  Trace  feature 

-  without  Trace  feature 

-  with  Simulator  instead  of  “golden"  computer 

with  predetermined  results 

In  order  to  facilitate  the  comparison  cf  comparable  cuantities,  the 
following  assumptions  are  made  concerning  each  projected  approach. 

1.  Each  (non-fiandom)  verification  program  has  on  the  average  25 
test  cases  per  instruction  resulting  in  approximately  5,000 
separate  test  cases  being  generated  for  an  architecture  of 
the  llIL-STD-1750  class. 

2.  A  programmer  productivity  of  2  test  cases  per  day  is 
assumed.  The  Air  Focce*s  suggested  rate  for  Operational 
Flight  Program  development  is  110  lines  cf  High  order 
Language  instruction  statements  per  month,  and  75  lines  of 
machine  language  instruction  statements  per  month.  They  are 
modified  for  calculating  the  development  cost  of 
verification  programs  based  on  the  following  reasons: 

a.  Verification  programs  are  greatly  simpler  in  complexity 
and  organization  than  operational  flight  programs. 
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b. 

Verification  programs  are 

easily  modularized 

into 

different  program  segments. 


c.  There  is  a  minimal  amount  of  inter-module  communication 
in  verification  programs. 

d.  Each  module  in  a  verification  program  contains  only 
very  simple  program  logic. 

e.  Each  module's  function  is  extremely  repetitive  in 
nature  (i.e.,  load  data  values,  perform  operation,  and 
check  result  with  the  expected  result) . 

f.  The  main  storage  area  (reserved  for  constants  and 
expected  results)  comprises  30  to  50  percent  of  each 
test  module. 

Therefore,  a  programmer  productivity  rate  of  200  lines  of 
High  Order  Language  instruction  statements  per  month,  180 
lines  of  machine  language  instruction  statements  for  control 
program  code,  and  280  lines  of  machine  language  instruction 
statements  for  test  program  code  per  month  are  used  in 
calculating  the  development  cost  for  the  verification 
programs. 

Software  maintenance  costs  are  projected  assuming  that  two 
errors  are  found  per  a  thousand  lines  of  delivered  code  and 
that  each  error  takes  1  man  week  to  correct.  Software 
maintenance  costs  are  based  cn  a  ten  year  life  of  the 
verification  program. 

Total  recurring  costs  are  based  cn  a  ten  year  life  of  the 
program  and  30  computers  being  certified. 

The  MIL-STD-1750  computer  has  a  32K,  16-bit  words  of  main 
storage. 

The  VAX  11/780  computer  system,  the  HIl-STD- 1553  I/O 
channel,  and  the  ES-232  I/O  channel  are  available  to  support 
the  Master/Slave  approach  at  mere  cost. 

The  time  required  to  perform  validation  does  net  include  the 
time  allocated  to  cabling  up  the  computer  and  verifying  the 
I/O  interface.  It  is  assumed  that  a  8-hour  time  slice  would 
be  more  than  adequate  to  support  this  activity. 

For  calculation  purposes,  the  unit  under  test  computer  is 
assumed  to  be  a  500  ROP  machine,  and  the  Master  computer  is 
capable  of  executing  one  million  instructions  per  second  and 
prints  at  a  300  line  per  minute  rate. 

A  cost  figure  of  $70F  per  man  year  is  used  in  developing 
dollar  cost  totals. 
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NOTE: 


Contrary  to  intuition,  the  type  of  program  knovn  as  a 
"Control  Program"  varies  both  in  si26  ana  function 
throughout  the  12  approaches  analyzed.  Therefore,  cost 
figures  for  each  approach  will  reflect  this  variance  between 
control  programs.  Similarly,  the  overhead  processing  per 
test  case  for  each  approach  varies  frcm  50  instructions  for 
AM/AYK-15A,  Diagnostic  and  FTP,  to  1,000  for  Lockstep, 
10,000  for  AVP  and  15,000  for  Fandom.  These  numbers  were 
developed  from  the  data  gathered  during  the  first  phase  of 
the  study  and  represent  the  type  of  processing  required  in 
the  test  case  initialization  and  execution  followed  by  the 
verification  of  results.  The  first  three  approaches  invoke 
in-line  tests,  thus  taking  little  overhead.  The  remaining 
approaches  require  operating  system  overhead,  or  control 
card  processing  (AVP) ,  or  simulation  and  incur  a  large 
amount  of  additional  processing. 
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"7.2.1  BK/AYK- 1 5A  ATP  in  a  Manoal  Test  Cent iouraticn 


7.2. 1.1  Description 


A  verification  program  developed  from  the  A!i/AYK-15A  ATP,  targeted  for 
a  Manual  Test  Configuration,  would  yield  a  satisfactory  and  thorough 
static  test  of  the  flIL-STD-1750  instruction  set.  Dsing  the  existing 
AK/AYK-15A  ATP  as  a  starting  point,  the  following  modifications  must 
he  made.  Each  test  nodule  must  be  analyzed  for  content  with 
irrelevant  test  cases  excluded  and  relevant  test  cases  added  to 
ircrease  coverage.  The  supervisor  program  must  also  he  modified  to 
communicate  with  the  newly  defined  I/O  interface  associated  with  the 
manual  test  configuration.  The  estinated  size  of  the  finished 
verification  program  is  S6K,  requiring  three  program  loads. 


7.2. 1.2  Non-Eecurring  Start-Op  Costs 


The  cost  of  implementing  a  verification  program  based  on  modifying  the 
AB/AYK-15A  ATP  under  a  Manual  Test  Configuration  would  consist  of  the 
following  software  development  components. 


Modification  of  AN/AYK-15A  ATP  =  3.1  man  years 

(Modify  30X  of  30K  EAL*  =  9K) 

(Write  IK  BAL  Control  Program) 

Source  Tape  Generator  Program  =  0.3  man  years 

(500  BAL) 

Test  Plan  Document  =  0.3  man  years 

TOTAL  3.7  man  years 

♦ - + 


|*BAL  means  Basic  Assembler  language) 

1  or,  equivalently,  | 

I  Machine  Language  Instructions  | 

I  HOL  means  High  Order  language  | 

I  I 

INote  that  where  HOL  and  EAl  figures) 
/appear  on  the  same  line,  this  means) 
(that  portions  of  the  program  will  ) 
Jbe  written  in  each  language.  ) 

♦ - + 
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■i.2.1.3  Recurring  Costs 


Ihe  recurring  costs  associated  with  a  verificaticn  afproach  based  on 
tie  irodificaticn  of  the  AN/AIK-15  A  ATP  under  a  Manual  Test 
Configuration  consist  of  two  major  components  -  verification  cost,  and 
the  cost  to  sustain  the  software.  The  recurring  costs  reflects  the 
cost  per  computer  tested,  assuming  30  computers  tested  over  a  10  year 
period.  The  verification  costs  are  prcpcrticnal  to  the  staffing 
allocated  during  each  vendor  verification  and  the  actual  time  it  takes 
tc  complete  each  verificaticn  procedure.  The  cost  tc  sustain  the 
software  based  on  the  number  of  source  lines  of  code  (not  program 
size)  are  calculated  in  the  following  table.  An  additional  cost, 
although  nominal  in  nature,  is  the  cost  of  generating  the  verification 
program  source  tape  for  the  vendor  to  make  IPL  tapes  from.  This  cost 
is  considered  part  of  the  verificaticn  cost.  Table  7-2  contains  a 
summary  of  recurring  costs. 


■/.  2.1.1'  Time  Required  tc  Perform  Validation 


The  time  required  to  perform  validation  using  a  verification  program 
lased  on  the  modification  of  the  AN/ATK-15A  ATP  under  a  Manual  Test 
Configuration  consists  of  the  summation  of  mount  times,  memory  load 
times  and  execution  times.  Assuming  5  minutes  to  mount  the  tape,  3 
minutes  to  load  memory  and  8  minutes  tc  process  the  test  modules  on 
each  load  tape  and  print  cut  the  results,  the  maximum  time  to  run  the 
verification  test  error  free  would  be  calculated  as  follows; 


Verification  time  =  Nj  ♦  5  minutes  to  mcunt  each  tape  + 

N2  ♦  3  minutes  to  load  and  go  + 

Nj  *  2  minutes  to  execute  the  program  and 
print  out  results 


where: 


Nt  =  number  of  tape  mounts  =  3  worst  case 

Ne  =  number  of  program  leads  =  3 


Verification  time  =  3*5+3*3+3*8 

=  48  minutes 
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1.7  5  Inpact  to  SEAFAC  Resources 


The  implenieDtation  of  a  verification  approach  based  on  the 
irodif ication  of  the  AN/AYK-15A  ATP  under  a  Manual  Test  Configuration 
wculd  require  the  use  of  MIL-STD-1750  support  software  (cross 
assembler,  linking  loader,  and  simulator)  on  the  development  computer 
system.  SEAFAC  personnel  would  develop  and  maintain  the  certification 
program  as  well  as  prepare  the  scurce  tape  to  give  tc  each  vendor. 
During  the  verification  process,  there  would  be  no  impact  on  the 
development  computer,  but  SEAFAC  personnel  would  be  required  to  assist 
in  the  integration,  initiation,  and  observation  of  the  verification 
program  executing  on  the  unit  under  test. 

The  cost  data  impact  to  SEAFAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7-2. 
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Table  7-2.  Cost  Summary  for  the  ftN/RYK-15A  ATP  Approach  Under  A 

Manual  Test  Configuration 

- - - - - - - — - - - 

I  i  Cost 


I  Man 

Item  JYear 

Mcn-Becurring  Start-Dp  Costs  | 

I 

Hardware  I 

-  Development  Computer  I  0  |  0 

i  I 

Software  i  I 

-  MIL-STD-1750  Support  Software  |  0  1  0 

(Cross  Assembler,  Linking  leader.  Simulator)  |  I 

-  Modification  of  AN/AYK-15A  ATP  |3-1  j217 

-  Source  Tape  Generation  Program  10.3  1  21 

I  1 

Other  I  I 

-  Test  Plan  Document  |0.3  |  21 

TOTAL  13.7  1259 

1  1 

Recurring  Costs/Computer  1  1 

1  1 

Hardware  1 

-  Maintenance  1  1  0 

1  I 

Software  1  1 

-  Maintenance  10.C521  3.7 

40K  ♦  2  errors/K  *  $1,400/30  1  1 

I  1 

Personnel  1  1 

-  Coverage  to  Observe  Execution  ard  Analyze  10.04  1  2.8 

Besults  (2  People  for  1  Reek)  1  1 

1  i 

Other  1  1 

-  Test  Plan  to  Vendor  with  Verification  Source  10.0041  0.28 

0.0961  6.78 


TOTAL 
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7.2.2  AK/AYK- 1 5A  ATP  in  a  Haster^/Slavc  lest  Configurat^cg 


7. 2. 2.1  Description 


A  verification  program  based  on  modifying  the  AN/AYK-15A  ATP,  targeted 
fcr  a  Master/Slave  lest  Configuration  mould  result  in  a  verification 
giccess  closely  resembling  one  based  on  the  FTP  or  Diagnostic 
approach.  The  AN/AYK-ISA  test  modules  mould  be  analyzed  for  content 
mith  the  irrelevant  test  cases  deleted  and  additional  test  cases  added 
tc  increase  coverage.  The  supervisor  program  mould  be  modified  to 
communicate  mith  the  control  program  on  the  Master  tc  facilitate  the 
loading  of  test  modules  into  the  unit  under  test  and  the  passing  of 
test  results  back  to  the  Master  for  recording.  The  control  program  on 
the  Master  mould  handle  all  I/O  and  mould  be  initially  invoked  by  a 
bootstrap  program  loaded  on  the  unit  under  test  (slave)  by  the  vendor 
at  the  start  of  the  verification  process.  The  approximate  size  of 
each  of  the  softmare  modules  folloms: 


Supervisor 

s 

8K 

mords 

Test  Modules 

= 

88K 

mords 

Control  Program 

s 

8K 

mords 

Bootstrap  Program 

s 

IK 

mords 

7. 2. 2. 2  Non-fiecurring  Start-Dp  Costs 


The  cost  of  implementing  a  verification  program  based  on  modifying  the 
AN/AYK-15A  ATP  under  a  Master/Slave  Test  Configuration  mould  consist 
of  the  folloming  software  development  components: 
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Hodification  of  AN/AYK-15A  ATP 
(Modify  30X  of  30K  EAL  =  9K) 
(irite  IK  BAL  Supervisor 
Program) 

Control  Program  (Master) 

(2K  HOL;  500  BAL) 

Bootstrap  Program 
(400  BAL) 

Source  Tape  Generator  Program 
(500  BAL) 

I/O  Test  Programs  (IK  BAL) 

Test  Plan  Document 

TOTAL 


3.1  nan  years 

1.1  nan  year 

0.3  nan  years 

0.3  man  years 

0.5  nan  years 
0.3  man  years 
5.6  man  years 


7. 2. 2. 3  Becurring  Costs 


Ite  recurring  costs  associated  «ith  modifying  the  AN/AYK-15A  ATP  to 
tun  under  a  Master/Slave  Test  Configuration  consists  of  the  usage  of 
the  Master  computer  (zero  cost)  and  the  staffing  for  the  integration, 
initiation  and  observation  of  the  verification  process,  plus  the 
analysis  of  the  results.  The  second  major  cost  is  the  cost  to  sustain 
all  software.  Table  7-3  contains  a  summary  of  recurring  costs. 


7.2. 2. 4  Time  Beguired  to  Perform  Validation 


The  time  required  to  perform  the  complete  verification  process  is 
calculated  as  fellows: 

Verification  time  =  Bootstrap  Load  and  Go  (5  minutes)  ♦ 

Master  Computer  Ccntrol  Program  + 
Initialization  (5  minutes) 

Verification  Program  Execution 

Time  (1  minute)  ♦ 

I/O  Transfer  Time  (3  seconds) 

=  11  minutes. 
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Table  7-3.  Cost  Summary  for  the  AN/AFK-15A  ATP  Approach  Under  A 
Haster/Slave  Test  Configuration 


Item 

Kcn-Fecurring  Start-Qp  Costs 
Hardvare 

-  Development/Master  Computer 

-  llIL-SID-1553  and  RS-232  1/0  Interfaces 

Software 

-  iIIL-SID-1750  Support  Software  (Cross  Assembler, 

Linking  Loader,  Simulator) 

-  Bootstrap  Load  Program 

-  Source  Tape  Generation  Program 

-  Control  Program  on  Master 

-  Modification  of  AN/AYK-15A 

Other 

-  Test  Plan  Document 

TOTAL 

Recurring  Costs/Computer 

Hardware 

-  Maintenance 

Software 

-  Maintenance 

44. 4K  *  2  errors/K  ♦  $1,400/30 
Personnel 

-  Coverage  to  Initialize,  Observe  and  Analyze 

Results  (2  People  for  1  Reek) 

-  Technician  to  Supervise  Integraticr  of  1/0 

Interface 

Otter 

-  Test  Plan  to  Vendor  with  Verification  Source 


Cost 


Ban 

Years 


0 

0 


0.3 

0.3 


0.5 

0.3 

5.  6 


0.059 

0.04 

0.004 

0.004 

0.107 


K  $ 


0 

0 


21 

21 

77 

217 

35 

21 

392 


4.  14 

2.8 

0.28 

0.28 

7.5 


TOTAL 
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7. 2. 2. 5  Impact  to  SEAFAC  Rescatces 


Th€  implementation  of  a  ^ecificaticn  program  by  modifying  the  existing 
AM/AYK-15A  ATP  to  cun  under  a  Haster/Slave  Test  Configuration  would 
utilize  the  support  software  (HIL-STD- 17 EO  cross  assemblers,  linking 
leader,  and  simulator)  as  well  as  normal  system  utilities  on  the 
Master  computer  daring  system  dewelopnent.  Daring  the  verification 
process,  the  Master  computer  would  play  a  passive  role  serving  as  an 
I/O  controller  for  the  verification  program.  SEAFAC  personnel  would 
be  reauired  to  develop  and  sustain  the  verification  program,  as  well 
as  the  bootstrap  and  control  programs.  During  testing,  SEAFAC 
personnel  must  also  supervise  the  integration  and  initialization  of 
the  verification  process. 

The  cost  data  impact  to  SEAFAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7-3. 


7,2.3  Random  Instruction  in  a  Manual  Test  Configuraticc 


7.2.3. 1  Description 


A  verification  program  developed  using  a  Fandom  Instruction  design 
philosophy  targeted  for  a  Manual  Test  Configuration  would  result  in  a 
verification  process  with  certain  severe  limitations  given  the  current 
memory  constraints  of  the  computer.  The  Random  tests  approach  in  a 
standalone  mode  consist  of  a  random  instruction  sequence  generator,  a 
supervisor  program  and  a  high  fidelity/quality  "golden"  simulator. 
The  random  instruction  sequence  generator  generates  a  sequence  of 
instructions  that  are  executed  by  the  hardware  and  simulated  by  the 
simulator  The  supervisor  program  then  compares  the  generated  and 
simulated  results  and  prints  out  the  results. 

The  approximate  size  of  each  software  component  is: 


Control  Program 

Random  Instruction  Generator 

"Golden"  Simulator 


4R  words 
24R  words 
300K  words 


Given  the  32K  memory  size  specification  for  all  units  under  test,  a 
dynamic  random  instruction  verification  pregram  under  a  Manual  Test 
Configuration  would  not  be  possible  to  implement.  An  alternative  is  a 
static  approach.  Bandom  sequence  of  instructions  are  generated 
offline  and  saved.  These  instruction  sequences  are  then  simulated  and 
their  results  saved.  load  tapes  consisting  of  a  supervisor  program, 
sets  of  random  instruction  sequences  and  test  results  are  generated. 
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H€  load  tapes  are  executed  on  the  unit  under  test.  Ibe  supervisor 
piogran  sequentially  executes  the  randcm  instructicn  test  sets, 
ccvparinq  generated  values  to  expected  values  and  printing  the 
results.  The  number  c£  randca  instructicn  sequences  executed  is 
limited  by  the  time  allccated  for  the  validation  process,  and  the  time 
it  takes  to  load  and  execute  the  test  cases. 

li  the  supervisor  program  takes  8K  words,  that  leaves  24K  words  for 
test  cases,  expected  results,  and  generated  results.  £ach  test  set  is 
(cn  the  average)  40  words  long,  the  expected  results  fcr  each  test  set 
is  60  words;  each  test  set  effectively  contains  10  executable  test 
cases.  In  the  24K  of  data  space,  240  test  sets  could  be  allocated. 
Therefore  to  facilitate  the  execution  cf  125,000  test  sets  with  a 
tctal  of  1,250,000  test  cases  being  executed,  521  separate  memory 
leads  would  have  to  be  made.  Furthermcre,  the  Air  Force  would  have  to 
supply  the  vendor  with  12.75  megawords  worth  of  data  to  generate  the 
lead  tapes.  ((521  loads  •  24K  words)  ♦  BK.) 

It  has  previously  been  stated  that  when  the  Fandom  Instructicn  method 
has  been  applied  to  verify  the  architecture  of  a  computer  which  is 
capable  of  executing  four  million  instructions  per  second,  that  most, 
if  not  all,  of  the  architectural  discrepancies  are  discovered  after 
cne  hour  of  processing.  In  erder  to  obtain  the  necessary  coverage, 
confidence,  and  completeness,  this  interprets  to  tbe  execution  of 
125,000  sets  of  randcm  instruction  sequences  with  each  having  an 
average  length  of  10  executed  instructions  (although  the  sequences  are 
32  instructions  in  length,  the  percentage  distribution  of  branch 
instructions  randomly  appearing  brings  the  average  number  of 
instructions  executed  per  test  set  sequence  tc  10).  Therefore, 
1,250,000  random  instructions  are  tested  per  hour  (on  the  four  million 
instructions  per  second  computer) ,  with  each  instruction  in  itself 
being  a  test  case.  The  term  "test  case",  therefore,  when  used  in 
ccnjunction  with  the  random  instructicn  approach,  refers  to  a  single 
instruction  which  is  generated,  and  verified. 

NOTF:  Tbe  execution  of  1,250,000  randomly  generated 

instruction  test  cases  has  been  judged  a  priori  to  be 
comparable  frem  a  quality  viewpoint  to  the  5,000 
manually  generated  test  cases  used  in  other  approaches. 


7.2. 3.2  Non-Recurring  Start-Up  Costs 

The  cost  for  implementing  a  verification  program  based  on  the  Random 
Instruction  approach  under  a  manual  test  configuration  would  consist 
of  the  following  software  development  costs: 
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Random  Instruction  Generator 
(6K  HOL;  IK  BAL) 

3.0 

man 

years 

Simulator 

(12K  HOL;  2K  BAL) 

S 

6.0 

man 

years 

Supervisor  Program 
(4K  BAL) 

S 

1.8 

man 

years 

Tape  Generator  Program 
(500  BAL) 

s 

0.3 

man 

years 

Test  Plan  Document 

= 

0.3 

man 

years 

TCTAL 

11.4 

man 

years 

*7. 2. 3. 3  Recurring  Costs 


Tie  recurring  costs  associated  with  the  Random  Instruction  approach 
ittcleaented  under  a  Manual  Test  Configuration  consists  of  three 
significant  components.  First,  the  cost  cf  generating  1,250,000  test 
cases  and  results  on  the  development  system  and  creating  the  test  case 
scurce  tapes  for  the  vendor,  pricr  to  in-house  testing.  Second  is  the 
personnel  requirements  necessary  to  observe  the  manual  test  procedure 
taking  place.  The  time  required  for  generating  the  1,250,000  test 
cases  and  results  based  on  15,000  instructions  being  ejcecuted  to 
generate  the  data  necessary  for  each  test  case  on  a  computer  (capable 
of  executing  one  million  instructions  pec  second)  is: 


1  sec 

15,000  *  1,250,000  instructions  • - -  instructicns  =  18,750  sec 

1,0C0,000 


Tlerefore  it  would  take  5  hours,  12  minutes  of  CPD  time  to  generate 
tie  data  to  be  put  on  a  single  tape  plus  the  I/C  transfer  time  for 
12.75  megawords.  The  cost  of  sustaining  the  certification  program  is 
the  third  component  of  recurring  ccsts. 


7. 2. 3. 4  Time  Required  to  Perform  Validation 


Using  the  formula  developed  previously,  the  verificaticn  time  to  load, 
execute  and  print  out  the  results  of  1,250,000  randomly  generated  test 
cases  would  be  as  follows: 
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Verification  tine  =  N|  *  5  ninutes  to  nount  each  tape  * 

Nj  ♦  3  minutes  to  lead  and  go  ♦ 

N2  *  2  minutes  to  execute  the  program  and 
print  out  results 


where: 


N|  =  number  of  tape  mounts  -  521  worst  case 

N2  =  number  of  program  loads  =  521 


Verification  time  =  521  ♦  5  ♦  521  *  3  ♦  521  *  2 

=  5210  min  =  86.8  hours  =  11  days  (8  hours/day) 


7. 2. 3. 5  Impact  to  SEiFAC  Besources 


Using  a  Manual  Test  Configuration  under  which  to  implement  a  Bandom 
Instruction  test  approach  places  a  significant  burden  on  SEAFAC 
personnel  to  be  present  during  the  11  days  of  testing.  Other 
reauirements  would  be  the  normal  support  software  (cross  assembler, 
linking  loader  and  simulation) ;  plus  the  development  of  the  random 
instruction  generation  program.  As  previously  mentioned,  the 
ceneraticE  of  the  random  instructions,  though  completely  automated 
requires  a  significant  amount  of  CFO  time  on  the  development  computer. 

The  cost  data  impact  to  SEAFAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7-tt. 
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Table  7-4.  Cost  Summary  for  the  Random  Instruction  Under  A 

Manual  Test  Configuration 


Item 


Cost 


Ban 

Years 


K  S 


Non-Recurring  Start-Op  Costs 
Hardvare 

-  Development  Computer 
Software 

-  MIL-STD-1750  Support  Software  (Cross  Assembler, 

Linking  Loader,  Simulator) 

-  "Golden"  Simulator 

-  Random  Instruction  Generation  Program 

-  Source  Tape  Generator  Program 

-  Supervisor  Program 

Otter 

-  Test  Plan  Document 

TOT  AL 

Recurring  Costs/Computer 

Hardware 

-  Bain te nance 

Software 

-  Baintenance 

25. 5K  •  2  errors/K  ♦  $1,400/30 
Personnel 

-  Coverage  to  Observe  Execution  and  Analyze 

Results  (1  Person  for  2  Reeks) 

Other 

-  Test  Plan  to  Vendor  with  Verification  Source 


6.0 

3.0 

0.3 

1.8 


0.3 

11.4 


420 

210 

121 

126 


21 

799 


0.034 

0.04 

0.004 

0.078 


2.33 


0.28 

5.46 


TOTAL 
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7.2.1J  Random  Instruction  in  a  Haster/Sla ve  Test  Confi onration 


7.2. 4.1  Description 


The  Random  Instruction  design  approach  is  ideally  suited  to  the 
Haster/Slave  Test  Configuration  for  implementing  the  fill-STD- 1 750 
verification  program.  Onder  this  approach,  the  random  instruction 
generator  and  "golden"  simulator  would  reside  on  the  Haster  computer. 
The  unit  under  test  (slave)  would  contain  supervisor  programs  whose 
functions  are  to  request  test  cases,  execute  them,  and  send  back  the 
results  to  a  control  program  running  in  the  Master  computer.  The 
control  program  supervises  the  generation,  simulation  and  comparison 
functions  on  the  Master  computer  and  legs  the  results.  The  Air  Force 
would  send  the  vendor  a  copy  of  the  bootstrap  program,  and  the  vendor 
would  bring  it  in  IPL  format  along  with  the  I/O  routines  at  the  time 
of  certification.  The  size  of  each  program  module  is  described  in  the 
previous  section. 


7. 2. 4. 2  Non-Becurring  Start-Dp  Costs 


The  costs  of  implementing  a  verification  program  based  on  the  Random 
Instruction  approach  under  a  Master/Slave  Test  Configuration  would 
consist  of  the  following  software  development  costs: 


Random  Instruction  Generator  (Master) 

(6K  HOL;  IK  BAL) 

3.0 

man 

years 

"Golden"  Simulator  (Master) 

(12K  HOL;  IK  BAL) 

= 

o 

1 

VO 

man 

years 

Ccntrcl  Program  (Haster) 

(3K  HOL;  IK  BAL) 

1.8 

man 

years 

Supervisor  Program  (Slave) 

(3K  BAL) 

= 

1.4 

man 

years 

Bootstrap  Program 
(400  BAL) 

= 

0.3 

man 

years 

Source  Tape  Generator  Program 
(500  BAL) 

= 

0.3 

man 

years 

I/O  Test  Programs  (IK  BAL) 

0.5 

man 

years 

Test  Plan  Document 

= 

0.3 

man 

years 

TCTJL 

13. e 

man 

years 
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7. 2. 4. 3  Eecurring  Costs 


The  recurring  costs  associated  with  the  Random  Instruction  approach 
under  the  Uaster/Slave  Test  Configuration  consist  of  ccsputer  usage  on 
the  Raster  computer  (zero  costs)  and  staffing  for  whatever  portion  of 
the  test  is  required,  especially  the  analysis  of  the  results-  This 
cost  is  augmented  by  system  integration  atd  initiation  costs.  The 
final  cost  component  is  the  cost  of  sustaining  the  software. 


7.2. 4- 4  Time  Required  to  Perform  Validation 


The  time  required  to  perform  the  complete  verificaticn  test  consists 
of  the  summation  of  Bootstrap  Load  and  Go  time.  Master  Computer 
Control  Program  Initiation  time.  Test  Case  Generation  and  Execution 
time,  and  I/O  Transfer  time.  An  estimate  of  these  times  follows: 

Test  Case  Generation,  Simulaticn  and  Verification 
Master 

15,000  instructions  per  test  case  overhead  to  generate, 
simulate,  and  verify  results  * 

1,250,000  test  cases  ♦ 

1  sec/1,000,000  instructions  (speed  of  master) 

Total  time  of  Generation/Verif icaticn  =  18,750  seconds  =  313  min 

Test  Case  Execution  cn  Slave 

1,000  instructions  pec  test  case  overhead  * 

1,250,000  test  cases  ♦ 

1  sec/500,000  instructions  (speed  of  slave) 

=  2,500  sec  =  42  min 

Bata  Transfer 

125,000  test  set  *  40  words  of  instructions  per 
test  set  + 

125,000  test  set  *  60  words  of  results  per 
test  set 

=  12.5  megawords  /  30K  words/sec  =  417  seconds  =  7  min 
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Verification  time  =  Bootstrap  load  and  Go  (5  minutes)  + 

Haster  Computer  Ccntrcl  Prcgram 

Initialisation  (5  minutes)  ♦ 

Test  Case  Generation  Time  (313  minutes)  ♦ 

Test  Case  Emecutior  Time  (42  minutes)  ♦ 

I/O  Transfer  Time  (7  minutes) 

=  6  hours,  12  minutes. 


NOTE:  The  major  component  in  the  verification  time  is  the 

time  to  generate,  simulate  and  verify  the  random 
instructions.  This  component  is  in  turn  proportional 
to  the  overhead  (15,000  instructions)  taken  to  perform 
this  task.  This  figure  assumes  a  10,000  to  1 
performance  ratio  on  the  simulator,  which  is  a  figure 
characteristic  of  simulators  developed  in  a  high  order 
language.  The  total  verification  time  is  a  worst  case 
analysis  and  makes  no  assumptions  regarding  potential 
speed  up  if  overlapped  processing  is  implemented. 
Total  time  is  improved  only  two  percent  if  a  parallel 
I/O  channel  is  used  instead  of  HIL-STE-1553  channel, 
and  would  be  degrated  by  17  percent  if  a  ES-232  channel 

is  used. 

« 


7. 2. 4. 5  Impact  to  SEAFAC  Besources 


A  verification  program  based  on  the  Fandom  Instruction  design 
philosophy  targeted  for  Master/Slave  Test  Configuration  would  utilize 
the  Haster  computer  both  during  the  development  phase  and  during  the 
verification  procedure.  SEAFAC  personnel  would  be  reouired  to  write 
all  software  modules  for  the  Raster  and  Slave.  This  would  include  the 
use  of  such  normal  support  software  available  on  the  Haster  (compiler, 
editor,  linking  loader)  as  well  as  HIl-STD-1750  support  software, 
(cross  assembler,  linking  loader,  and  simulator).  Luring  the  actual 
verification  process,  a  certain  portion  of  the  Haster  computer's 
computational  power  must  be  dedicated  to  the  generation,  simulation, 
verification  and  documentation  of  the  test  cases. 

The  cost  data  impact  to  SEAFAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7-5. 
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Table  7-5.  Cost  Summary  for  the  Pandcm  Instruction  Approach  Under  A 

Master/Sla ve  Test  Configuration 


♦  - 

1 

1 

1 

1 

Item 

1  Cost  1 

, - 1 

1  Man  1  1 

1  Years  j  K  $  j 

1 

1 

Non-Pecurring  Start-Up  Costs 

1 

1 

1 

1 

t 

1 

Hardware 

1 

1 

1 

i 

1 

-  Development/Master  Computer 

1  0 

0  1 

1 

-  ilIL-STL-1553  and  RS-232  I/O  Interfaces 

1  0 

0  1 

1 

1 

Software 

1 

! 

1 

1 

-  hIL-STD-1750  Support  Software  (Cross  Assembler,!  0 

0  1 

1 

Linking  Loader,  Simulator) 

i 

1 

1 

-  Bootstrap  Load  Program 

1  0.3 

21  ! 

1 

-  Source  Tape  Generation  Program 

1  0.3 

21  1 

1 

-  Control  Program  on  Master 

j  1.8 

1 26  i 

1 

-  "Golden"  Simulator 

i  6.0 

420  1 

1 

-  Supervisor  Program  on  Slave 

1  1.4 

98  1 

1 

-  Eandom  Instruction  Generation  Program 

j  3.0 

2  10  1 

1 

-  I/O  Test  Programs 

1  0.5 

35  1 

1 

1 

Other 

1 

1 

1 

1 

1 

-  Test  Plan  Document 

1  0.3 

21  1 

1 

1 

TOTAL 

1  13.6 

9  52  1 

1 

1 

1 

Pecurring  Costs/Computer 

1 

1 

1 

1 

1 

1 

Hardware 

1 

1 

1 

1 

-  Maintenance 

1  0 

0  1 

I 

1 

Software 

1 

1 

I 

1 

1 

-  Maintenance 

1  0.039 

2.7  1 

1 

1 

28. 9K  ♦  2  errors/K  ♦  $1,400/30 

1 

1 

1 

1 

1 

Personnel 

1 

1 

r  1 

1 

-  Coverage  to  Initialize,  Observe  and  Analyze 

i  0.04 

2.8  1 

1 

Results  (2  People  for  1  Peek) 

1 

1 

-  Technician  to  Supervise  Integration  of  I/O 

1  0.004 

o 

• 

00 

1 

Interface 

1 

1 

1 

1 

Other 

1 

1 

1 

1 

1 

-  Test  Plan  to  Vendor  with  Verification  Source 

j  0.004 

0.23  1 

I 

1 

+- 

TOTAL 

i  0.087 

6.061 
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7.2.5  AVP  Id  A  Test  Conf iquratiop 


'/.2.5.1  Description 


J  verification  progran  developed  using  ar.  AVP  design  philosophy 
targeted  for  a  Manual  Test  Corf iguraticn  vould  force  mcdularization  in 
the  following  manner.  The  verification  pregram  will  consist  of  an  8K 
supervisor  program  and  a  24K  data  buffer.  The  supervisor  program's 
rcle  is  to  process  test  cases  resident  in  the  data  buffer,  calling  the 
vendor's  print  routine  to  output  results  as  they  are  generated.  Test 
cases  will  be  made  up  of  four  types  of  statements  (80  character 
records) : 

a.  ID  statement  -  classifying  the  type  cf  test. 

b.  Set  up  statements  -  specifying  the  initialization 
values. 

c.  Execute  statement  -  specifying  the  instruction  sequence 
to  be  executed. 

d.  Result  statements  -  specifying  the  expected  values  to 
be  compared  to  the  generated  results. 

Test  cases  will  average  about  10  statements  in  length  occupying  400 
werds  of  memory.  A  library  system  resident  on  the  development  system 
would  be  used  to  handle  the  test  case  data  base.  The  library  system 

allows  edit  and  file  capabilities-  The  24K  test  case  data  buffer 
would  then  be  large  enough  tc  contain  an  average  cf  61  test  cases. 
Therefore,  there  would  have  to  be  82  distinct  program  loads  under  the 
manual  test  configuration  to  process  the  5000  expected  test  cases 
enerated.  The  test  cases  would  occupy  2  million  words  of  auxiliary 
torace.  It  should  be  ncted  that  the  large  amount  of  storage  required 
or  this  approach  is  due  tc  the  overhead  involved  ir  having 
0-character  control  cards  of  which  cnly  ten  to  twenty  percent 
contains  meaningful  information.  This  implies  that  ceitain 
implementation  strategies  could  compress  unused  blanks,  although  (for 
this  analysis)  a  tradeoff  like  this  will  net  be  considered. 

The  vendor  would  be  responsible  fer  the  generation  of  the  load  tapes 
necessary  to  handle  82  loads.  The  Air  Force  would  be  respcnsinle  for 
sending  the  2  megawords  of  data  to  the  vendor. 


7. 2. 5. 2  Non-Recurring  Start-Dp  Costs 


The  cost  for  implementing  a  verification  program  based  on  the  AVP  test 
approach  would  be  strictly  software  development  costs.  The  cost 
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breakdown  is  as  follows: 


Case  Develocment 

25  test  cases  per  instruction  *  200  instructions 
2  test  cases  per  day  productivity 
250  days/man  year 

Supervisor  Program  De velcpaent 

(3K  Machine  Language  Instructions) 

library  System  on  Develcfment  Computer 

Tape  Generation  Program  on 
Development  Computer 
(500  Machine  Language  Instructions) 

Test  Plan  Document 

TOTAL 


7.2. 5. 3  Recurring  Costs 


Tfce  recurring  costs  associated  with  the  AVP  approach  under  the  Manual 
Test  Configuration  are  proportional  to  the  staffing  allocated  during 
the  vendor  certification  process,  and  the  time  it  takes  to  complete 
the  certification.  A  second  cost,  though  nominal  in  nature  is  the 
cost  of  generating  the  test  case  tape  for  the  vendor  prior  to  starting 
the  in-house  test  procedure.  The  final  recurring  cost  component  is 
the  cost  to  sustain  the  software. 


7.2. 5. 4  Time  Required  to  Perform  Validation 


The  time  required  to  perform  the  entire  certification  consists  of  the 
Eunmaticn  of  mount  times,  memory  load  times,  and  execution  times. 
Assuming  5  minutes  to  mount  the  tape,  3  minutes  to  load  memory  and  2 
winutes  to  process  the  61  tests  cases  in  each  program  load  and  print 
out  the  results.  The  maximum  time  to  run  the  certification  tests 
error  free  would  be  calculated  as  follows; 


5,000  test  cases 
2,500  days 

10  man  years 

1.3  man  years 

N. C. 

O. 3  man  years 

0.3  nan  years 
11.9  man  years 


-35 


e  176175& 


FINAL  REPCEl 


Felruary  29,  1980 


Verification  time  =  N,  *  5  minutes  to  mount  each  tape  ♦ 

Na  ♦  3  minutes  to  load  and  go  + 

Nj  *  3  minutes  to  execute  the  program  and 
print  cut  results 


where: 


Nt 

=  number 

of 

tape  mounts  = 

82  worst  case 

Nz 

=  number 

of 

program  loads  = 

82 

Verification  time 

=  82  * 

5  ♦ 

82  ♦  3  ♦  82  *  3 

=  902  minutes  =  15  hours,  2  minutes 


7. 2. 5. 5  Impact  to  SEAFAC  Resources 


The  implementation  of  the  AVP  approach  under  a  Manual  Test 
Configuration  would  require  the  use  of  a  development  computer  with 
litrary  system,  and  related  MIL-STD-1750  support  software  (cross 
assembler  and  simulator) .  The  development  computer  would  also  be 
necessary  to  generate  the  source  tapes  to  give  to  the  vendor.  There 
would  be  no  impact  to  the  development  computer  during  the  in-house 
certification  process.  SIAFAC  personnel  would  be  required  to  develop 
and  maintain  the  test  cases  as  well  as  the  supervisor  program. 

The  cost  data  impact  to  SEAFAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7-6. 
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Table  7-6.  Cost  Summary  for  the  SVP  Approach  Onder  A  HaBual  Test 

Configuration 


Item 

Non-Pecurring  Start-Op  Costs 
Hardware 

-  Development  computer 
Software 

-  ilIL-STD-1750  Support  Software  (Cress  Assembler, 

Linking  Loader,  Simulator) 

-  Test  Cases 

-  Supervisor  Program 

-  Library  System  on  Development  Computer 

-  Source  Tape  Generation  Program 

Ctber 

-  Test  Plan  Document 

TOTAL 

Pecutring  Costs/Computer 

Hardware 

-  maintenance 

Software 

-  maintenance 

Supervisor  Program:  3K  ♦  2  errors/K  ♦  $1,400/30 
Test  Cases:  2,000K  *0.1  errors/K  *  $1,400/30 

Personnel 

-  coverage  to  Observe  Execution  and  Analyze 

Eesults 

Other 

-  Test  Plan  to  Vendor  with  Verification  Source 


Cost 


man 

Years 


10.0 

1.3 

0 

0.3 


0.13b 


0.004 


0.18 


0,2  3 


12.58 


TOTAL 
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7.2.6  a VP  in  a  Master/Slave  Test  Conf igcration 


7.2.6. 1  Description 


A  verification  progran  developed  using  an  AVP  design  philosophy 
targeted  for  a  Haste r/Slave  Test  Configuration  Mould  result  in  a 
ccitpletely  autoaated  verification  approach.  The  verification  prograa 
Mould  be  segmented  into  an  8K  Supervisor  Program  with  a  24K  test  case 
data  buffer.  A  bootstrap  program  would  communicate  with  a  control 
program  on  the  Hasten  computer  to  first  load  the  Supervisor  program 
into  the  unit  under  test  (Slave) .  The  supervisor  would  then 
automatically  roll  in  test  cases  and  output  results  as  the  test  cases 
Mere  executed.  Test  case  size  and  layout  would  be  the  sane  as  in  the 
Hanual  Test  Con’figaration  and  are  described  previously. 

The  library  system  on  the  Hasten  computer  Mould  be  used  to  develop  the 
test  cases  and  handle  their  selection  for  processing  by  the  control 
program  running  during  the  certification  process.  The  amount  of  data 
to  be  transferred  between  Hasten  and  Slave  is  as  folloMs: 

Test  Cases  2, COOK  words 

Supervisor  8K 

Pesults  5K 


TOTAL  2,013K  words 

Tie  Air  Force  would  send  the  vendor  a  copy  of  the  bootstrap  program, 
and  the  vendor  would  bring  it  in  lEL  format  at  the  time  of 
certification. 


7.2. 6.2  Mon-Becurring  Start-Dp  Costs 


The  costs  for  implementing  a  verification  program  based 
test  approach  and  an  automatic  test  configuration  would 
software  development  costs.  The  cost  breakdown  parallels 
test  configuration  as  follows: 


on  the  AVP 
be  strictly 
the  manual 
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Test  Case  Development 


5,000  test  cases  at  2  tests  cases  per  day 

Control  Program  Development  (Uastec) 

(2K  HOL  Instructions; 

500  Uachine  Language  Instructions) 

Bootstrap  Program  Development  (Slave) 

(400  Machine  Language  Instruct icns) 

Supervisor  Program  Development  (Slave) 

(3K  Machine  Language  Instructicns) 

library  System  on  Master  Computer 

Source  Tape  Generation  Program 

(500  Machine  language  Instructicns) 


10.0  man  years 
1.3  man  years 

0.3  man  years 
1.  1  man  years 

N.C. 

C.3  mac  years 


I/C  Test  Programs  (IK  BAL) 


C.5  man  years 


Test  Plan  Document 


0.3  man  years 


TOTAL  13.8  man  years 


7. 2. 6. 3  Becurring  Costs 


The  recurring  costs  associated  with  the  AVP  approach  under  the 
Master/Slave  Test  Configuration  are  associated  with  system  integration 
and  initialization.  These  costs  should  not  be  greater  than  1  man  day 
tctal.  The  other  major  costs  would  be  attributed  to  staffing  during 
the  vendor  certif icaticn  process  and  the  cost  of  sustaining  the 
verification  software.  A  minimal  cost  due  to  generating  the  source 
tape  for  delivery  to  each  vendor  is  also  incurred. 


7.2. 6. 4  Time  Eequired  to  Perform  Validation 


The  time  required  to  perform  the  ccmplete  verification  test  consists 
of  the  summation  of  Bootstrap  Load  and  GO  time.  Master  Computer 
Control  Program  Initiation  time.  Test  Case  Execution  time  and  I/O 
Transfer  time.  An  estimate  of  these  times  are  as  follows: 
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lest  Case  Execution  line 

5,000  test  cases  ♦  10,000 
instructicns/test  case 

overhead  =  5,000,000  iDSti.uctioiis 

5,000,000  instructiors/500, 000 

instructions/sec  =  10  seconds 


I/O  Transfer  Tine 

2013k  words  /  30K  words/sec  (over  KII-STE-1553)  =  67  seconds 

Verification  Time  =  Bootstrap  load  and  Go  (5  minutes)  ♦ 

Master  Computer  Ccntrol  Program  Initialization 
(5  minutes)  ♦ 

Test  Case  Eiecution  Time  (10  seconds)  ♦ 

Print  Results  (90  seconds)  ♦ 

I/O  Transfer  Time  (67  seconds) 

=  12  minutes,  h7  seconds. 


7. 2. 6. 5  Impact  to  SEAFAC  Resources 


Tte  implementation  of  the  AVP  approach  under  a  Master/Slave  Test 
Configuration  would  rely  heavily  on  the  Master  computer  during  the 
developrment  phase  orf-  the  verification  program,  and  during  the  actual 
running  of  the  certification  process.  The  Master  computer  would  need 
to  support  a  library  system  as  well  as  the  usual  selection  of  support 
software  (assembler,  linking  loader  and  simulator).  SFAFAC  personnel 
would  be  required  to  develop  and  modify  the  test  cases,  bootstrap 
supervisor  and  control  programs. 


The  cost  data  impact  to  SEAFAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7-7. 
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Table  7-7.  Cost  Suamaty  for  the  AVP  Apcrcach  Under  A  tJaster/Slave 

Test  Configu ration 


♦ ' 

1 

1  Cost 

1 

1 

Man 

1 

Itea 

Years 

1  K  $ 

1 

Ncn-Eecurring  Start-Up  Costs 

1 

1 

Hardware 

1 

-  Dewelopaent/liaster  Computer 

0 

1  0 

1 

-  MIL- STD- 1553  and  RS-232  1/C  Interfaces 

0 

1  0 

1 

1 

Software 

1 

-  MIL-STD-1750  Support  Software  (Cross  Assembler, 

0 

i  0 

1 

Linking  Loader,  Simulator) 

1 

-  Bootstrap  Load  Program 

0.3 

1  21 

1 

-  Source  Tape  Generation  Program 

0.3 

1  21 

1 

-  Control  Program  on  Master 

1.3 

1  91 

1 

-  lest  Ccises 

10.0 

1700 

1 

-  Supervisor  Program 

1.1 

1  77 

1 

-  Library  System  on  Master/Cevelopment  Computer 

0 

1  0 

1 

-  I/O  Test  Program 

0.  5 

i  35 

1 

1 

Other 

1 

-  Test  Plan  Document 

0.3 

1  21 

1 

1 

1 

TOTAL 

13.8 

1  966 

1 

1 

1 

Recurring  Costs/Computer 

1 

1 

Hardware 

i 

-  Maintenance 

0 

1  0 

1 

1 

Software 

1 

-  Maintenance 

0.  143 

1  10.02 

1 

1 

Control  Program:  7.4K  ♦  2  errcrs/K  *  $1,400/30 
Test  Cases:  2,000K  *0.1  errors/k  *  $1,400/30 

1 

1 

Personnel 

I 

-  Coverage  to  Initialize,  Observe  and  Analyze 

0.04 

1  2.3 

1 

Results  (2  People  for  1  Reek) 

1 

-  Technician  to  Supervise  Integration  of  I/O 

0.004 

1  0.28 

1 

Interface 

1 

1 

Other 

1 

-  Test  Plan  to  Vendor  with  Verification  Source 

0.004 

1  0.28 

1 

1 

TOT  AL 

0-191 

1  14.38 
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7.2.7  Diaqpostic  Modification  in  a  Manual  Test  ConfiQ^:ratio£ 


7.2.7. 1  Description 


A  verification  program  based  on  modifying  a  diagnostic  program  <- 
programs  for  the  MIL-STD-1750  to  run  under  a  Manual  Test  Configuration 
closely  resembles  the  end  results  from  modifying  an  existing  FTP  into 
a  certification  program  as  is  described  subsequently.  The  development 
process  would  require  the  analysis  of  an  existing  diagnostic  program 
or  programs,  to  determine  the  amount  of  coverage  each  instruction  is 
given.  '  The  next  step  would  be  to  remove  implementation  dependent 
code,  add  supplemental  test  cases,  and  modify  the  control  structure  of 
the  diagnostic  program  to  facilitate  dynamic  display  of  results  on  nhe 
printer. 

The  verification  approach  would  be  modularized  according  to  the 
initial  modularization  of  the  diagnostic  program  selected  for 
modification.  A  completely  new  modularization  scheme  could  be  imposed 
oriented  to  a  supervisor  module  and  several  test  modules,  each 
containing  test  cases  verifying  a  certain  type  of  instruction  or 
architectural  feature  of  the  machine.  The  estimated  size  of  the 
verification  program  is  96K  words,  thus  requiring  three  distinct 
program  loads.  The  Air  Force  would  be  responsible  for  making  the 
source  available  to  the  vendor  for  his  in-house  generation  of  IPL 
tapes. 

A  risk  factor,  associated  ^ith  the  modification  of  different  vendor's 
diagnostic  programs,  concerns  the  potential  incompatibility  of 
assembler  language  grammar  formats,  thereby  requiring  a  translation 
process  to  occur. 


7. 2. 7. 2  Non-fiecurring  Start-Up  Costs 


The  cost  of  implementing  a  certification  program  based  on  the 
modification  of  existing  diagnostic  cregram  or  programs  under  a  Manual 
Test  Configuration  would  consist  of  the  following  software  development 
components; 


Ciagnostic  Modification 

(Modify  18K  BAL;  Write  5.5K  BAl) 

7.0 

man 

years 

Tape  Generator  Program 

= 

0.3 

man 

years 

Test  Plan  Document 

= 

0.3 

man 

years 

TOTAL 

7.6 

man 

years 
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1.2. 7. 3  Hecucring  Costs 


IVe  recurring  costs  associated  with  the  diagnostic  modification  under 
a  Ranual  Test  Configuraticu  ace  pcoporticnal  to  the  staffing  allocated 
during  the  vendor  verification  process,  and  the  actual  time  xt  takes 
to  complete  the  verification.  A  second  cost,  though  nominal  in  nature 
is  the  cost  of  generating  the  verification  program  source  tape  for 
each  vendor.  The  third  component  is  the  cost  of  sustaining  the 
verification  software  programs. 


7. 2. 7. 4  Time  Required  tc  Perform  Validaticr 


The  time  required  to  perform  the  entire  verification  can  be  calculated 
using  the  formula  developed  previously. 

Verification  time  =  Nt  *  5  minutes  to  mount  each  tape  ♦ 

Nj  *  3  minutes  to  load  and  go  ♦ 

N2  ♦  8  minutes  tc  execute  the  program  and 
print  out  the  results 

where: 

Ni  =  number  of  tape  mounts  ®  3  worst  case 

Nj  =  number  of  program  loads  =  3 

Tltr^iicre  V  fetixica  Li^,  L  Lime  =  ♦  1^2  *  3  *  S 

=  48  minutes 


7-2. 7. 5  Impact  to  SEAFAC  Resources 


The  implementation  of  a  verificaticn  program  based  on  the  modification 
of  an  existing  diagnostic  program  or  programs  to  run  under  a  Ranual 
Test  Configuration  would  require  the  use  of  MIL-STE- 1750  support 
cftware  (cross  assembler,  linking  loader,  and  simulator)  on  the 
evelcpment  computer  system.  SEAFAC  personnel  would  develop  and 
maintain  the  certification  program  as  well  as  prepare  the  source  tape 
tc  give  to  each  vendor-  During  the  verificaticn  process,  there  would 
te  nc  impact  on  the  development  computer,  but  SEAFAC  personnel  would 
he  required  to  assist  in  the  integration,  initiation,  and  observation 
cf  the  verification  program  executing  on  the  unit  under  test. 

The  cost  data  impact  tc  SEAFAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7-8. 
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Table  7-8.  Cost  SuDmary  for  the  Ciagnostic  Approach  Doder  A  uanual 

Test  Conf igoratioE" 


+  - 

1 

I  Cost 

1 

1 

1 

1  tian 

1 

Item 

1  Years 

K  $ 

1 

Non-Necurring  Start-Dp  Costs 

1 

1 

1 

Hardware 

J 

1 

1 

1 

-  Development  Computer 

1  0 

1 

0 

1 

1 

Software 

1 

1 

-  NIL- SID- 1750  Support  Software  (Cross  Assembler 

,  1  0 

0 

1 

Linking  Loader,  Simulator) 

1 

1 

-  Diagnostic  Program  Modification 

1  7-  0 

490 

1 

-  Source  Tape  Generation  Program 

JO.  3 

1 

21 

1 

1 

Other 

1 

1 

1 

-  Test  Plan  Document 

JO. 3 

21 

1 

1 

TOTAL 

J  7.  6 

532 

1 

1 

1 

Pecurring  Costs/Computer 

1 

1 

1 

1 

1 

Hardware 

1 

1 

-  Maintenance 

1  0 

1 

0 

1 

I 

Software 

1 

1 

1 

-  Maintenance 

1 0. 053 

3.7 

1 

40K  X  2  errors/K  *  f1,40C/30 

1 

1 

1 

1 

Personnel 

1 

1 

1 

-  Coverage  to  Observe  Ixecution  and  Analyze 

1  0.  04 

2.8 

1 

Results  (2  People  for  1  Week) 

1 

1 

1 

1 

Other 

1 

1 

1 

-  Test  Plan  to  Vendor  with  Verification  Source 

JO. 004 

0.28 

1 

1 

•f  - 

TOTAL 

Io.097 

b.7e 

♦ 
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7.2.8  Diagnostic  aodif ication  in  a  Master/Slave  Test  Configuration 


1 


".2.6.1  Description 


A  verification  program  based  on  nodifyirg  a  diagnostic  program  or 
picgramc  for  the  aiL-STD-1750  under  a  Haster/Slave  Test  Configuration 
would  cli.  'tiy  parallel  a  verification  program  based  on  tne 
"edification  of  an  existing  FTP  to  run  under  a  'las ter/Sia ve  Test 
(  c I  f  1  Cl u z  r. t i  I  z c  ic  i  rt  ly  l  rrmerit 

would  require  tne  analysis  of  the  diagnostic  programs  so  that  the 
pioper  test  cases  could  be  added  (to  increase  the  coverage  of  the 
test)  and  deleted  (to  eliminate  implementation  dependencies  inherent 
to  all  diagnostic  programs).  The  control  structure  of  the  diagnostic 
program  would  also  nave  to  be  modified  tc  facilitate  communication 
with  the  tlaster  computer. 

"  ie  approximate  size  of  the  verification  program  would  be  96K,  divided 
into  ’4  supervisor  program  and  test  modules  as  described  previously. 
Tie  supervisor  program  would  communicate  with  a  control  program  or  the 
“aster  cemputer,  requesting  test  modules  to  be  transferred  and  sending 
bach  test  results.  The  ccntrcl  program  on  the  ("aster  computer  would 
ro’;tc  output  messages  from  the  unit  under  "-est  to  a  hardcopy  device, 
and  load  test  modules  frem  auxiliary  storage  to  send  tc  the  unit  under 
test  upon  request.  A  bootstrap  load  program  would  also  have  to  be 
develope  «itr.  the  source  being  made  available  to  the  vendor  prior  to 
ir- house  testing. 


7.2.  6. 2  Non-r  jou,.  . .  .to  :  ‘-a:  f-op  Costs 


The  cost  of  imple.fM-n  ting  a  verificaticr  approach  based  on  the 
mcdification  of  existing  aTL-STD-17f0  diagnostic  pregrams  uuaer  a 
naster/Slave  Test  Configuration  would  consist  of  the  fo..lowir.u 
software  development  components: 
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Eiagnostic  Nodif ication 

(Modify  18K  BftL;  Write  5.5K  Eftl) 

Control  Prograa  (Master) 

(2K  HOL;  500  BAL) 

Bootstrap  Program 
(400  faAL) 

Bounce  Iap«  Generator  Program 
(500  EAL) 

I/O  Test  Program  (IK  BAL) 

Test  Plan  Document 

TOTAL 


7.0  man  years 

1.1  man  year 

0.3  man  years 

0.3  man  years 

0.5  many  years 
0.3  man  years 
9.5  man  years 


".2.8.3  Recurring  Costs 


Tie  recurring  costs  associated  vith  modifying  a  diagnostic  program  to 
tun  under  a  Master/Slave  Test  Configuration  consist  of  computer  usage 
cn  the  Master  computer  (zero  cost)  -and  staffing  for  the  integration, 
initiation,  observation  of  the  verification  program  and  analysis  of 
the  results.  A  second  component  is  the  cost  of  sustaining  the 
verification  program. 


7.2. 6.4  Time  Reguired  tc  Perform  Validation 


The  time  requited  to  cerfors  the  ccsplete  verification  process  is 
calculated  by  finding  the  summation  of  the  Bootstrap  Load  and  Go  time, 
faster  Computer  Control  Program  Initiaticr  time.  Test  Case  Execution 
time  and  I/O  transfer  time.  An  estimate  of  these  tines  follows: 
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Verification  Program  Execution 

Plus  Logging  Results  =  30  sec 

I/O  Transfer  Time 

S6K  words  /  30K  word/sec 

(over  MIL-STD-1553)  =  3  sec 

Verification  tiae  =  Footstrap  Load  and  Go  (5  minutes)  + 

Master  Computer  Ccntrol  Program 
Initiali2at icn  (5  minutes)  + 

Verification  Prcgran  Execution 
Time  (30  minutes) 

I/O  Transfer  Time  (3  seconds) 

=  40  minutes,  3  seconds. 


7. 2. 8. 5  Impact  to  SEAFAC  Resources 


The  implementation  of  a  verification  program  by  modifying  existing 
diagnostic  programs  to  run  under  a  Master/Slave  Test  Configuration 
would  utilize  the  support  software,  (cross  assembler,  linking  loader, 
and  simulator)  during  program  development.  Curing  the  actual 
verification  process,  the  Master  computer  would  be  necessary  to  handle 
the  control  program  function  as  well  as  the  documentation  of  the  test 
results.  SEAPAC  personnel  would  be  required  to  develop  and  modify  the 
verification  program,  bootstrap  and  control  programs. 

The  cost  data  impact  to  SEAFAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7-9. 
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Table  7-9.  Cost  Sumniary  for  the  Eiagcostic  Approach  Under  A 
Master/Sla ve  Test  Cent iguration 


Item 

lion -■Recurring  Start-Op  Costs 
Hardware 

-  Development/Master  Computer 

-  aiL-STD-1553  nnd  RS-232  I/O  Interfaces 

Software 

-  MIL-STD- 1750  Support  Software  (Cress  Assembler, 

Linking  Loader,  Simulator) 

-  Bootstrap  Load  Program 

-  Source  Tape  Generation  Program 

-  Control  Program  on  Master 

-  Diagnostic  Modification 

-  I/O  Test  Program 

Other 

-  Test  Plan  Document 

TOTAL 

Fecurring  Costs/Computer 

Hardware 

-  Maintenance 

Software 

-  Maintenance 

44. 4K  *  2  errors/K  *  $1,460/30 
Personnel 

-  Coverage  to  Initialize,  Observe  and  Analyze 

Results  (2  People  for  1  Week) 

-  Technician  to  Supervise  Integration  of  I/O 

Interface 

Other 

-  Test  Plan  to  Vendor  with  Verification  Source 


Cost 


1  Man  1 

1  Years ) 

- 1 

1 

K  $  1 

t  1 

1  1 

1  i 

1  0  1 

1 

1 

1 

0  1 

i  0  t 

0  1 
i 

1  i 

1  0  I 

1 

0  1 

i  1 

10.3  i 

1 

21  1 

10.3  1 

21  1 

11.1  1 

77  1 

17.0  1490  1 

10.5  1 

i  1 

35  1 

1 

1  1 
JO. 3  1 

1 

21  1 

1  i  1 

19.5  1665  1 

1  1  1 

1  i  1 

1  1 

1  1 

1  0  1 
i  1 

0  1 

1  1 

10.059  I 

1  1 

4.  141 

o 

• 

o 

2.8  1 

1  1 

10.0041 

1  1 

0.23  1 

i  I 

1  1 

10.0041 

o 

• 

CD 

1  1 
10.107) 

7.5  1 

. 

TOTAL 


m 
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".2.9  FTP  in  a  Manaal  Te^  Conf igurat ion 


7.2.9. 1  Description 


S  verification  program  based  cn  the  FTP  design  philosophy  targeted  for 
a  Manual  Test  Configuration  could  be  develcted  in  twc  different  ways, 
he  first  is  to  use  an  existing  MIl-STE-17fO  FTP  to  modify  its  control 
tructure  to  facilitate  dynanric  display  cf  results  on  the  printer  and 
tc  analyze  and  augment  its  test  cases  to  achieve  25  test  cases  per 
instruction  average.  The  second  approach  is  to  implement  an  FT?  type 
program  through  the  entire  software  develcpment  process. 

This  type  of  verificaticn  program  would  he  divided  into  a  supervisor 
module  and  several  test  modules.  Each  test  module  would  contain  test 
cases  verifying  a  certain  type  of  instructicn  or  architectural  feature 
cf  the  machine.  The  supervisor  program  is  designed  and  coded  under 
the  assumption  that  no  instructions  work.  This  is  implemented  through 
utilizing  a  small  core  set  of  instructicns  to  handle  control  and 
slcwly  introducing  more  instructicns  as  confidence  in  their 
credibility  is  established  (center  cut  approach) .  The  supervisor 
program  invoices  each  test  module  and  outputs  test  results. 

The  estimated  size  of  the  supervisor  and  test  modules  is  S6K,  thus 
requiring  3  distinct  ,  program  loads.  The  Air  Force  would  be 
responsible  for  aaking  this  source  available  to  the  vendor  for  his 
ic-house  generation  of  the  IPL  tapes. 


7.2. 9.2  Non-fiecurring  Start-Dp  Costs 


The  cost  of  impleaectring  a  certif icaticn  crcgram  based  cn  the  FTP  test 
approach  would  depend  on  whether  or  not  the  program  development  was 
based  on  the  modification  of  an  existing  FTP.  The  software  cost 
breakdown  is  as  fallows; 
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FTP  Modification  = 

(Modify  of  30K  lines  of  EAl  =  9K) 

(Write  IK  BAL  lines  for  Contrcl  Program) 

3.  1 

man 

years 

FTP  Full  Eevelopment  = 

(40K  BAL  instructions) 

12.0 

man 

years 

Tape  Generator  Program  = 

(500  BAL  instructions) 

0.3 

man 

years 

Test  Plan  Documient  = 

0.3 

man 

years 

TCTAL 

3.7 

12.6 

man 

nan 

years/ 

years 

7. 2. 9. 3  Becarring  Costs 


The  rscurring  costs  associated  with  the  FTF  approach  under  a  Manual 
Test  Configuration  are  proportional  trc  the  staffing  allocated  during 
the  vendor  verification  process  and  the  actual  tine  it  takes  to 
complete  the  verification.  A  second  cost,  though  nciinal  in  nature, 
is  that  of  generating  the  FTF  source  tape  for  the  vendor.  Ike  cost  of 
sustaining  the  verification  software  programs  is  the  third  component. 


7.2. S.h  lime  Eeguired  to  Perform  Validation 


The  time  required  to  perform  the  entire  verification  consists  of  the 
summation  of  mount  times,  memory  load  times  and  execution  times. 
Assuming  5  minutes  to  mount  the  tape,  3  minutes  to  load  memory  and  8 
minutes  to  process  the  test  modules  on  each  load  tape  and  print  out 
the  results,  the  maximum  time  to  run  the  verification  test  error  free 
would  he  calculated  as  fellows; 
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Verification  time  =  Nj  *  5  minutes  tc  nscunt  each  tape  + 

Nj  *  3  minutes  tc  lead  and  gc  + 

Ng  *  8  minutes  to  execute  the  program  and 
print  out  results 


where : 


Nj  =  number  of  tape  mounts  =  3  worst  case 
Nj  =  number  of  program  loads  =  3 


Verification  time  =  3*5t3*3+3*8 

=  48  minutes 


7.2. 9. 5  Impact  to  SERFAC  Resources 


The  implementation  of  the  FTP  approach  under  Manual  Test  Configuration 
wculd  require  the  use  of  MIL-STD-1750  support  software  (cross 
assembler,  linking  loader  and  simulator) .  The  development  computer 
wculd  also  be  necessary  to  generate  the  source  tape  tc  give  to  the 
vendor.  There  would  be  no  impact  to  the  development  computer  during 
the  in-house  certification  process.  SFAFAC  personnel  would  be 
required  to  develop  and  maintain  the  certification  program. 

The  cost  data  impact  tc  SEAFAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7-10. 
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Table  7-10.  Cost  Summary  for  the  FTP  Approach  Order  A  Manual  Test 

Configuration 


Item 

Non-Becurring  Start-Op  Costs 
Hardware 

-  Developmeat  Computer 
Software 

-  MIL-STD-1750  Support  Software  (Cross  Assembler, 

Linking  Loader,  Simulator) 

-  FTP  Modification 

-  ftp  Full  Development 

-  Source  Tape  Generation  Program 

Other 

-  Test  Plan  Document 


TCTAl  BY  KCDITYING  FTP 
TOTAl  BY  EEVEICPING  NEH  FTP 

Becurring  Costs/Computer 

Hardware 

-  Maintenance 

Software 

-  Maintenance 

40K  ♦  2  errors/K  ♦  $1,400/30 
Personnel 

-  Coverage  to  Observe  Execution  and  Analyze 

Eesults  (2  People  for  1  Week) 


Other 

-  Test  Plan  to  Vendor  with  Verification  Source 


- 

Cost  I 


Man 

Years 


K  $ 


I 


0 


0 


0 


0 


3.  1 
12.0 
0.3 


217 

840 

21 


0.3 


21 


3.7  1259 

12.6  1382 


0 


0 


0.0531  3.7 


0.04  )  2.8 


0.0041  0.28 

0.0S7I  6.78 


TOTAL 
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7.2.10  FTP  in  a  Haste r/Slave  Test  Con f iguratior. 


7.2.10.1  Description 


A  certification  program  developed  using  a  FTP  design  philosophy 
targeted  for  a  Master/Slave  Test  Configuration  would  result  in  a 
ccirpletely  automated  certification  approach.  The  certification 
program  would  be  approximately  96K  in  size  and  be  divided  into  a 
supervisor  module  and  several  test  modules.  Each  test  module  contains 
test  cases  verifying  a  certain  type  of  instruction  or  architectural 
feature.  The  supervisor  program  would  be  designed  using  the  center 
out  approach  as  previously  described.  The  supervisor  program  would 
communicate  with  a  control  program  resident  on  the  Easter  computer, 
requesting  test  modules  to  be  transferred,  and  sending  baeJr  test 
results.  Ibe  control  pregrauBr  on  the  Master  computer  would  route 
cutout  messages  from  the  unit  under  test  to  a  hardcopy  device,  and 
pull  in  test  modules  from  auxiliary  storage  to  send  to  the  unit  under 
test  upon  request.  The  certification  program  could  he  developed  in 
two  different  ways.  The  first  is  to  take  an  existing  FTP,  and  modify 
its  control  structure  to  communicate  with  the  control  program  on  the 
Master  computer.  The  second  approach  would  be  to  implement  an  FTP 
type  certification  program  through  the  entire  software  development 
process.  A  bootstrap  load  program  would  also  have  to  be  developed 
with  the  source  being  made  available  to  the  vendor  prior  to  in-house 
testing. 


7.2.10.2  Ncn-Becurring  Start-Op  Costs 


The  cost  of  implementing  a  certification  approach  based  on  the  FTP 
test  approach  would  depend  on  whether  or  not  the  program  development 
was  based  on  the  modification  of  an  existing  FTP  (or  whether  one  is 
available  to  modify)  .  The  software  cost  breakdown  is  as  follows: 
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FTP  ilodif ication 

(Modify  30*  of  30K  lines  of  BAl  =  9K) 
(Write  IK  BAL  lines  for  Control  Program) 

3.  1 

man 

years 

FTP  Full  Development 
(UOK  BAL  instructions) 

= 

12.0 

man 

years 

Control  Program  Development  (Easter) 

(2K  HOL;  500  BAL  instructions) 

= 

1.1 

man 

years 

Bootstrap  Program  Development  (Slave) 

(UOO  BAL  instructions) 

= 

C.3 

man 

years 

Source  Tape  Generator  Program 
(500  BAL  instructions) 

= 

0.3 

man 

years 

I/C  Test  Programs  (IK  BAl  instructions) 

= 

0.5 

man 

years 

Test  Plan  Document 

TOTAL 

0.3 

5.6 

11^.5 

man 

man 

man 

years 

years/ 

years 

T.2.10.3  Recurring  Costs 


Ihs  recurring  costs  associated  with  an  FTP  approach  under  the 
Raster/Slave  Test  Configuration  are  associated  with  system  integration 
and  initiation.  These  costs  should  not  be  greater  than  1  man  day 
total.  The  other  costs  would  be  attributed  to  staffing  during  the 
vendor  certification  process,  and  the  cost  for  sustaining  the 
verification  software  programs. 


7.2.  10. h  Time  Required  to  Perform  Validation 


The  time  required  to  perform  the  complete  verification  tests  is 
calculated  by  finding  the  summation  of  the  Footstrap  Lead  and  Go  time. 
Raster  Computer  Program  Initiation  time.  Test  Case  Execution  time  and 
I/O  Transfer  time.  An  estimate  of  these  times  ate  as  fellows; 
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Test  Case  Execution  Time 

5,000  test  cases  *  50 

instructions/test  case  =  250,000  instructions 

250,000  instructions  /  500, CCO 

instructions/ second  =  0.5  seconds 

I/O  Transfer  Time 

96K  words  /  30K  wcrds/second 

(via  aiL-STD-1553)  =  3.0  seconds 

Verification  time  =  Bootstrap  load  and  Go  (5  minutes)  ♦ 

Master  Computer  Control  Program  Initialization 
(5  minutes)  ♦ 

Validation  Execution  Time  (0.5  seconds)  ♦ 

I/O  Transfer  Time  (3.0  seconds) 

=  10  minutes,  3.5  seconds. 


7.2.10.5  Impact  to  SEAFAC  Eesources 


The  implementation  of  the  FTP  approach  under  a  Haster/Slave  Test 
Configuration  would  utilize  the  support  software  resident  on  the 
Master  computer  during  program  development,  and  during  the  actual 
verification  process  the  Master  computer  would  be  necessary  to  handle 
the  control  program  function  as  well  as  documentation  of  test  results. 
SEAFAC  personnel  would  be  required  to  develop  and  modify  the 
verification  program,  bootstrap  and  control  program. 

The  cost  data  impact  to  SE^^FAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7>11. 
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Table  7-11.  Cost  Summary  for  the  FTP  Rtprcach  Under  A  Master/Slave 

Test  Configuration 


1 

Man 

Item 

1 

Years 

i  K  $ 

Non-Pecurring  Start-Up  Costs 

1 

1 

Harduare 

1 

1 

-  Development/daster  Computer 

1 

0 

1  0 

-  iJIL-SID-1553  and  ES-232  I/C  Interfaces 

1 

0 

1  0 

Soft Hare 

! 

1 

-  aiL-SID-1750  Support  Software  (Cross  Assembler,! 

0 

1  0 

Linking  Loader,  Simulator) 

1 

-  Bootstrap  Load  Program 

1 

0.3 

1  21 

-  Source  Tape  Generation  Program 

1 

0.  3 

1  21 

-  Control  Program  on  Master 

1 

1.1 

1  77 

-  FTP  Modification 

1 

3.  1 

1217 

-  FTP  full  Development 

1 

12.0 

1840 

-  I/O  Test  Programs 

1 

0.5 

1  35 

Other 

1 

1 

-  Test  Plan  Document 

1 

o 

f 

UJ 

1  21 

TOTAL  BY  KCCIFYING 

1 

FTP  / 

5.6 

1  392 

TOTAL  BY  DEVElCflFG  NEK 

FTP  1 

14.  5 

1  1015 

Pecurring  Costs/Computer 

1 

1 

1 

Hardware 

1 

1 

-  Maintenance 

1 

0 

i  0 

Software 

1 

i 

-  Maintenance 

1 

0.059 

1  4. 

14 

44. 4K  ♦  2  errors/K  ♦  SI, 400/30 

1 

Personnel 

1 

1 

K 

-  Coverage  to  Initialize,  Observe  arc  Anal 

! 

0.04 

1  2. 

8 

Results  (2  People  for  1  Keek) 

i 

-  Technician  to  Sunervise  Irtegraticr,  of  :i 

0. 0C4 

1  0. 

28 

Interface 

* 

Other 

i 

-  Test  Plan  to  Vendor  with  Verification  Source  j 

0.004 

1  0. 

28 

Cost 


j  0.107 


7.5 
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7.2.11  lockste p  in  a  Manual  Test  Configuration 


7.2.11.1  Description 


The  Lockstep  approach  is  not  suitable  for  impleaertaticn  under  a 
Manual  Test  Configuration.  A  verification  program  developed  using  the 
Lockstep  philosophy  could  be  implemerted  if  certain  major 
modifications  were  made  to  the  initial  aprroach.  Two  approaches  wiii 
he  described  in  this  section;  a  semi-Lcckstep  approach  hhere  tne  trace 
interrupt  is  present  (henceforth  referred  to  as  the  SLT  apprcach) ,  and 
a  seni-Lockstep  approach  where  the  trace  interrupt  is  not  present 
(henceforth  referred  to  as  the  SLNT  approach) .  Ecth  approaches 
execute  sequences  of  instructions  referred  to  as  test  buckets  with 
trace  information  being  stored  sequential  in  a  data  buffer  area. 

The  trace  interrupt  collects  the  following  information: 

BOfiCS 


16  General  Purpose  Registers  16 
Instruction  Counter  1 
Status  Word  1 
Fault  Register  1 
Interrupt  Mask  1 

TCTAL  20  words 


Dnder  the  SLT  approach,  the  trace  information  is  automatically 
collected  and  saved  by  the  hardware.  Dnder  the  SLNT  approach,  the 
trace  information  is  generated  through  a  software  implemented  trace 
routine.  when  a  sequence  of  instructions  has  finished  execution  the 
generated  trace  information  is  compared  against  predetermined  trace 
information.  Each  approach  contains  the  following  modules: 


-  Supervisor 

=  EK 

words 

-  Test  buckets 

5,000  tests  cases 

*  10  words/test  =  5CK 

words 

-  Predetermined  Trace 
50,000  cases  *  20 

Results 

words  trace  information  =  1,000K 

words 

-  Trace  Buffer  Area 

=  IK 

words 

The  SL.'iT  has  tne  additional  requirement  of  having  a  software  trace 
module.  The  supervisor  program  executes  the  test  tucket,  compares  the 
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Generated  trace  information  to  the  expected  trace  irfcrmation,  ar d 
prints  out  the  results. 

The  size  or  the  trace  buffer  area  directly  affects  the  size  of  the 
test  tuckets,  and,  the  number  of  test  buckets  that  can  be  contained  in 
ere  program  load.  The  fcllcwing  equation  expresses  this  relationship: 

Memory  size  =  Supervisor  size  ♦  Trace  Buffer  size  ♦ 

Test  Bucket  area  +■  Predetermined  Trace  Information 

^itere : 

Test  Bucket  area  =  Number  of  Test  Buckets  ♦  Test  Bucket  Size 
Predetermined  Trace  Information  =  Number  of  Test  Buckets 

*  Test  Bucket  Size  ♦  20 

Trace  Buffer  Size 

Test  Bucket  Size  - - - - 

20  words  per  instruction 

For  a  32K  machine,  we  are  proposing  an  8K  supervisor  and  a  IK  trace 
buffer. 

let  NTB  =  Number  of  Test  Buckets 

TBS  =  Test  Bucket  Size 

Therefore: 


1024  words 

•tge  - - 

20  words  per  instruction 
=  51  instructions  per  bucket 

And  substituting  back  into  the  top  equation 

32K  =  8K  +  IK  +  NTB  *  51  +  NTB  *  5 1  *  20 


23K  =  NTD  *  51  (1  +  20) 
Sclving  for  NTB: 


23K 

NTB  - -  =  22  test  cases 

50  (21) 

It  should  be  mentioned  that  certain  architectural  features  require  a 
set  sequence  of  instructions  to  be  executed  in  order  to  verify  the 
ctoper  execution.  Therefore,  the  size  of  the  test  buckets  must  allow 
for  this  factor.  Analysis  from  data  gathered  on  various  test 
approaches  suggest  that  minimal  test  bucket  size  should  be  no  less 
than  30  words. 
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Stf  cost  cr  iO-tienieGtir.  .3  a  c*-rtificati< 
SINT  acprcdcti  t^ouia  consist  of  the  folli 

Test  Case  Development 

5,000  test  cases  *  10  words/ 
test  case 

*^0,000  words/(51  words/test 
b  uc  ke  t ) 

1  test  racket  ♦  results/ 
day  productivity 
150  days/year 

Supervisor  Program  Develcpnent 
(4K  EAL  instructions) 

Software  Trace  Module  Development 
(850  BAL  instructions) 

Tape  Generator  Program  for  Source 
(500  BAL  instructions) 


Test  Program  Development 


SLT  TOTAL 
SLNT  TOTAL 
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"’.r.ll."’  Ffecutriii^  Costs 


" tf  rprurrirg  co^ts  associated  with  the  £11  and  SLNT  approaches  are 
r  I  cr  or  t  i ''r  ai  to  tne  staffing  allocated  during  verification  process, 
ard  the  time  it  taKes  to  conclete  the  verification.  A  second  cost  is 
attrihuted  to  generating  the  source  load  tape  for  each  vendor's  IPL 
tare  Generation.  Tne  third  cost  ccmrcrert  is  the  cost  of  sustaining 
tte  verification  software. 


.  2  .  1 1 .  u  line  Beqaired  to  Ferform  Validation 

h  e  tiire  required  to  perforn  the  entire  ver  i  f  ica  ti  cr.  car  be  calculated 
i5inc  the  tclr owing  forntla: 

Verification  tine  =  t  *  time  tc  mount  IPL  tape  ♦ 

Nj  •  tiie  tc  lead  nemcry  start  program  ♦ 

Nj  *  time  to  execute  program  load 


where: 

N,  =  number  cf  tape  ocunts 
Nj  =  rum her  of  pregran  leads. 

hssutrirg  minutes  to  meant  a  tape,  3  minutes  to  load  th«.  memory  and 
start  the  piogiam  and  2  minutes  to  process  the  22  test  hucXets  in  each 
ricGiam  loau  ana  print  cut  the  results.  The  maximum  time  to  run  the 
v6rification  occurs  vher  N ,  =  Nj  (separate  mount  required  for  each 

pioaram  load).  The  verification  time  tc  complete  an  error  free 
fxecG^icn  ol  the  complete  program  would  he: 
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Verification  time  =  Ni  *  5  minutes  tc  mount  each  tape  + 

N2  *  3  minutes  tc  3cad  and  go  + 

Na  *  2  minutes  to  execute  the  program  and 
print  cut  results 


where: 


Nj  =  number  of  tape  mounts  =  46  worst  case 
N2  =  number  of  program  loads  =  46 


Verification  time  =  46  *  5  ♦  46  ♦  3  +  46  *  2 

=  460  minutes  =  7  hours,  40  minutes 


7.2.11.5  Impact  to  SEAFAC  Resources 


The  implementation  of  the  SLT  or  SINT  approach  would  require  the  use 
of  a  development  computer  preferably  with  a  library  system  to 
facilitate  the  test  bucket  data  base.  Support  software  consisting  of 
cross  assembler,  linking  loader  and  simulator  must  also  be  available. 
The  development  computer  would  be  used  tc  generate  the  source  tapes  to 
give  to  the  vendor.  There  would  be  no  impact  to  the  development 
computer  during  the  in-house  verification  process.  SIAFAC  personnel 
would  be  required  to  develop  and  maintain  the  test  buckets  as  well  as 
the  supervisor  program. 

The  cost  data  impact  to  SEAFAC  (and  the  previously  discussed  cost 
cata)  is  summarized  in  Table  7-12. 
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Tatle  7-12.  Cost  Summary  for  the  Lockstep  Approach  Cnder  A  Manual 

Test  Configuration 


- -  4 

Cost 


Item 


Han 

Years 


K  J 


Non-Pecurring  Start-Up  Costs 

1 

1 

Rardware 

-  Development  Computer 

1 

1 

|0 

1 

1 

1 

1  0 

Software 

-  MIL-STD-1750  Support  Software  (Cross  Assembler, 

Linking  Loader,  Simulator) 

-  Test  Cases  with  Results 

-  Supervisor  Program 

-  Software  Trace  Program 

-  Source  Tape  Generation  Program 

1 

1 

1  0 

1 

|4.0 

1  1.6 
jO.  4 
10.3 

1 

1 

1  0 

1 

1  280 

1  126 

1  28 

1  21 

Other 

-  Test  Plan  Document 

1 

1 

jO.  3 

1 

1 

1  21 

TOTAL  KITH  EAPDPAEE  TRACE 
TOTAL  WITH  SCFTPAPE  TRACE 

|6.4 
16.  8 

14  48 
j  476 

Recurring  Costs/Computer 

1 

1 

1 

1 

J 

1 

Hardware 

-  Maintenance 

1 

1 

|0 

1 

1 

1  0 

Software 

-  Maintenance 

Supervisor;  4.5K  *  2  errcrs/K  *  $1,400/30 

Test  Cases:  1,050K  ♦  0.2  errots/K  ♦  $1,400/30 

1 

1 

)0.  146 
1 

1 

1 

1 

1  10.22 

1 

1 

Personnel 

-  Coverage  to  Observe  Execution  and  Analyze 
Results  (2  People  for  1  Peek) 

1 

1 

10.04 

1 

1 

1 

1  2.8 

1 

Ot  her 

-  Test  Plan  to  Vendor  with  Certification  Source 

1 

1 

1  0 . 004 
1 - 

i 

i 

i  0.2S 

1 - 

0.19 


13.3 


TOTAL 
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7.2.12  Lockstep  in  a  Haster/Sla ve  Con  figuration 


7.2.12.1  Description 

A  verification  program  developed  using  a  Lcckstep  design  philosophy  is 
ideally  suited  for  a  Master/Slave  Test  Configuration.  This  approach 
can  be  separated  into  four  different  implementations.  First,  the 
master  computer  can  be  a  "golden"  KII-ETC-1750  ccmouter.  Second,  a 
MIl-STD-1750  simulator  running  on  the  development  machine  can  be  a 
"golden"  computer.  Third,  the  MIL-STD-17E0  under  test  may  cr  may  not 
have  the  trace  interrupt  feature.  The  compensation  for  lack  of 
hardware  trace  has  been  discussed  previcusly,  and  it  is  assumed  that  a 
similar  software  module  can  be  developed  in  this  situation.  A  fourth 
possible  variation  on  this  approach  can  occur  if  we  assume  that  no 
simulator  or  "golden"  computer  exists.  Instead,  predetermined  results 
are  calculated  during  test  case  development  and  these  results  are 
ccmpared  to  the  generated  trace  results.  This  variation  closely 
resembles  the  semi-Lockste p  approaches  (SLT  and  ELNT)  described 
previously.  All  approaches  execute  sequences  of  instructions  referred 
tc  as  test  buckets  with  trace  information  stored  sequentially  in  a 
data  buffer  area.  The  trace  data  collected  will  consist  of  20  words 
reflecting  the  state  of  the  computer  after  the  execution  of  the 
current  instruction  (see  Section  7.2.11.1  for  details).  When  a 
sequence  of  instructions  has  finished  execution,  the  trace  information 
generated  by  the  unit  under  test  (slave)  is  compared  with  the  trace 
information  generated  by  the  "golden"  computer  or  simulator,  except  in 
tbe  sen  i-Lockstep  approaches  where  the  generated  trace  data  is 
ccmpared  to  the  expected  trace  data.  The  results  are  then  printed 
cut. 

The  verification  program  based  on  a  Lcckstep  approach  under  a 
Master/Slave  Test  Configuration  would  be  divided  into  the  following 
module  j: 

Footstrap  Load  Program  =  IK  words 

Supervisor  =  8K  words 

-  Test  Bucket  Data  Base  =  50K  words 

-  Predetermined  Results  (optional)  =  1,OOOK  words 

Simulator  (optional)  =  300K  words 

Control  Program  =  8K  words 


I/O  Test  Programs 


4K  words 


n  t-i 


1 
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The  following  sequence  of  events  describes  the  interaction  and 
function  of  each  module.  (Note:  the  semi-Lockstep  approach  is 

described  later.) 


1.  The  bootstrap  load  program  establishes  communication  with 

the  control  program  on  the  Master  computer  and  starts  the 

transfer  of  a  supervisor  program.  This  process  takes  place 
for  the  unit  under  test  as  well  as  the  "golden"  computer,  if 
applicable. 

2.  The  supervisor  program  requests  test  buckets  for  execution 
from  the  Master  computer. 

3.  The  unit  under  test  executes  the  test  buckets,  while  the 

"golden"  computet  or  the  simulator  executes  the  same 

instructions. 

h.  The  results  from  the  unit  under  test  and  the  "golden" 

computer  or  the  simulator  are  sent  hack  to  the  Master 
computer  for  cemparison. 

5.  The  results  are  recorded. 


6.  Steps  1  through  5  are  repeated  until  all  test  buckets  and 
results  have  been  transmitted,  executed  and  compared. 

Per  the  semi-Lockstep  approach  there  is  no  "golden"  computer,  or 
simulator  and  the  sequence  of  events  would  flow  in  similar  order  with 
the  cemparison  of  generated  results  tc  the  predetermined  expected 
results  being  made  by  the  control  program  on  the  Master  computer. 


he  following  is  a  breakdown  of  data  transfers  taking  place  via  the 
cntrcl  program: 


Supervisor 

to 

Unit  Under  Test 

= 

8K 

words 

Test  Cases 

to 

Unit  Under  Test 

= 

50  K 

words 

Eesults  from 

Unit  Under  Test 

o 

o 

o 

words 

Supervisor 

to 

"Golden"  Computer 

= 

8K 

words 

Test  Cases 

to 

^en"  Computer 

= 

50  K 

words 

Fesults  from 

"GolQeii"  Computer 

= 

1 ,000K 

werds 

TOTAL 

2, 116K 

words 
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7.2.12.2  Non-Eecurring  Start-Op  Costs 


The  cost  components  for  implementing  a  verification  approach  based  on 
a  Lockstep  test  approach  under  a  Master/Slave  Test  Configuration  would 
depend  upon  the  final  variation  of  the  test  approach  selected.  The 
software  development  cost  would  be  as  fcllcws: 


Control  Program  (Master) 

(3K  HCL;  IK  BAL) 

1.8 

man 

years 

Footstrap  Program  (Slave) 

(400  BAL) 

0.3 

man 

years 

Supervisor  Program 
(3K  HAL) 

1.4 

man 

year 

Test  Buckets 

= 

3.0 

man 

years 

Test  Bucket  Results  (optional) 

1.0 

man 

year 

Software  Trace  Module  (optional) 
(850  BAL) 

= 

0.4 

man 

years 

Simulator  (optional) 

(12K  HOL;  2K  BAL) 

S 

6.0 

man 

years 

Source  Tape  Generator  Program 
(500  BAL) 

= 

0.3 

man 

years 

I/O  Test  Programs  (IK  BAL) 

0.5 

man 

years 

Test  Plan  Document 

- 

0.3 

man 

years 

The  Non-Recurring  Start-Op  costs  for  the  "golden"  ccmputer  can  he 
broken  down  into  the  hardware  procurement  cost  and  the  architecture 
verification  process  performed  to  validate  the  integrity  of  the 
"golden"  computer.  This  verification  is  extremely  crucial  in  assuring 
the  quality  of  the  system  and  would  require  detailed  analysis  almost 
ccmparable  to  any  of  the  other  approaches. 

The  estimates  for  these  costs  are  as  fcllcws; 

HIL-SID-1750  Computer  Acquisition  and 

Ground  Support  Equipment  =  J500K 

"Golden"  Computer  Verification  =  2.5  man  years 
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The  software  totals  then  for  each  approach  are  as  fcllcws; 

lockstep  with  "Golden”  Computer  =  10- 1  man  years 

lockstep  with  Simulator  =  13.6  man  years 

Semi-Lockstep,  predetermined  Results  =  8-6  man  years 

Note:  An  additional  0-4  man  years  would  be  added  to  each 

total  if  a  software  trace  feature  was  required  to  be 
developed  and  integrated. 


7-2-12-3  Recurring  Costs 


The  recurring  costs  associated  with  the  lockstep  approach  and  its 
variations  consists  of  the  hardware  maintenance  cost  associated  with 
the  "gclden"  computer,  the  personnel  allocated  during  the  verification 
process,  and  the  cost  of  sustaining  the  verification  s'^ftware 
programs. 


7.2.12.4  lime  Required  to  Perform  Validation 

The  time  required  to  transfer,  execute  and  compare  the  5,000  test 
tuckets  can  be  calculated  by  substituting  appropriate  cons-ants  into 
the  following  equation: 

Verification  time  =  Footstrap  load  Time  for  Dnit  Under  Test  + 

Footstrap  Load  Time  for  "Gclden"  Computer  + 
Master  Computer  Initialization  Time  ♦ 

Test  Case  Transfer  Time  to  Unit  Under  Test  + 
Test  Case  Transfer  Time  to  "Golden"  Computer  ♦ 
Test  Case  Execution  Time  + 

Supervisor  Program  Overhead  ♦ 

Test  Results  Transfer  from  Unit  Under  Test 
to  Master  + 

Test  Results  Transfer  from  "Gclden"  Computer 
to  Master  + 

Time  to  Compare  Trace  Information  and  Print 
Results- 
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Substituting  line-by-line  we  have: 

Verification  time  =  5  min  + 

5  min  ♦ 

5  min  ♦ 

2  sec  ♦ 

2  sec  ♦ 

(50,000  instructions  /  500,000  instructions/ 
sec  =  0,1  sec)  ♦ 

{50K  instructions  *  IK  overhead  factor  / 
500K  instructicns/sec  =  100  sec)  ♦ 

33.3  sec  + 

33.3  sec  + 

10  min 

Total  Time  =  27  minutes,  50  seconds. 


7.2.12.5  Impact  to  SEAFAC  Resources 


The  implementation  of  the  Lockstep  approach  for  a  verification  program 
under  a  Master/Slave  configuration,  would  reguire  the  use  of  a  library 
system  on  the  Master  computer  to  control  the  development  of  test 
tuckets  and  potentially  their  predetermined  results.  Support  software 
consisting  of  a  cross  assembler,  linking  loader  and  simulator  would 
also  have  to  be  available  on  the  Master.  SEAFAC  personnel  would  be 
required  to  develop  and  maintain  the  test  buckets  and  all  software 
modules  that  comprise  the  verification  program  as  well  as  supervise 
the  certification  process  each  time  a  vendor  brings  a  box  to  be 
tested.  If  a  "golden”  computer  is  selected,  then  SEAEAC  personnel 
must  also  develop,  verify  and  maintain  it,  and  its  related  Ground 
Support  Equipment.  Also  additional  space  and  power  must  be  made 
available  to  accommodate  the  "golden"  computer. 

The  ccst  data  impact  to  SEAFAC  (and  the  previously  discussed  cost 
data)  is  summarized  in  Table  7-13. 
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Table  7-13.  Cost  Summary  for  the  Lcckstep  Approach  Onder  a 
Master/Slave  Test  Ccc f iguratior. 


J  Cost 

1 

1 

1  Ban 

1 

Item 

1  Years 

K  $ 

1 

Non-Becurring  Start-Op  Costs 

1 

1 

Barduare 

1 

1 

- 

Development/Waster  Computer 

i  0 

0 

1 

- 

HIL-STD-1553  and  ES-232  1/C  Interfaces 

i  0 

0 

1 

- 

"Golden"  Computer 

1  7.14 

500 

) 

i 

Software 

1 

1 

1 

1 

- 

MIL- SID- 1750  Support  Software  (Cross  Assembler, 

1  0 

0 

1 

Linking  Loader,  Simulatcr) 

1 

1 

- 

Bootstrap  Load  Program 

1  0.3 

21 

1 

- 

Source  Tape  Generation  Program 

1  0.3 

21 

1 

- 

Control  Program  on  Faster 

J  1.8 

126 

1 

- 

Supervisor  Program 

1  1>9 

98 

1 

- 

Test  Buckets 

1  3.0 

21C 

1 

- 

Test  Bucket  Besults 

1  1.0 

70 

1 

- 

Software  Trace  Module 

1  0.4 

28 

1 

- 

Simulator 

I  6.0 

420 

1 

— 

I/O  Test  Programs 

1  0.5 

J 

35 

1 

1 

Other 

1 

1 

( 

1 

- 

Test  Plan  Document 

1  0.3 

21 

1 

- 

"Golden"  Computer  Verification 

1  2.5 

175 

1 

t 

TOTAL  WITH  "GCIDIN"  CCHPDTEE 

1  17.  2 

1207 

1 

1 

TCTAl  RITB  SIMULATOF 

113.6 

943 

1 

TOTAL  WITH  PEEEETEFMINED  EESDLTS 

1  8.6 

602 

1 

(NOTE:  ADD  $28K  TC  EACP  FIGURE  IF 

1 

1 

NO  HAEDFAPE  TRACE  IS  AVAILABLE) 

i 

1 

+ 
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Table  7-13.  Cost  Summary  for  the  Lockstep  Approach  Order  a 
Ma.‘iter/Sla ve  Test  Configuration  (cent) 


Item 

1  Man 
i Years 

1  K  $ 

Recurring  Costs/Computer 

1 

Hardware 

1 

-  Maintenance 

1  0 

1  0 

-  ‘'Golden"  Computer  Maintenance 

i0.086 

1  6.0 

Software 

1 

i 

-  Maintenance 

1 

"Golden"  Computer:  58K  •  2  errors/K  ♦  11,400/30 

10.077 

1  5.41 

Simulator;  7  IK  ♦  2  errors/K  *  $1,400/30 

10.096 

1  6.72 

Predetermined  Results; 

1 

1,051K  *  0.2  errors/K  *  $1,400/30 

|0.  14 

i 

i  9.81 

Personnel 

1 

1 

-  Coverage  to  Initialize,  Observe  and  Analyze 

i  0. 04 

1  2.8 

Results  (2  People  for  1  Week) 

1 

-  lecnnician  to  Supervise  Integration  of  I/O 

10.004 

J  0.28 

Interface 

1 

Other 

1 

\ 

-  lest  Plan  to  Vendor  with  Verification  Source 

10.004 

1  0.2S 

"GCIDER"  CCHPOTEE  TOTAL 

10.211 

114.77 

SIBDLATOE  TOTAL 

j  0. 144 

1 10.08 

PREDETERMINED  RFSDLTS  TOTAL 

1 0. 188 

1 13. 17 

Cost 
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7.3  COALITY  OF  VERIFICATION  APPROACHES  ANP  SOFTHAEE  VALIDATICK  COSTS 


A  discussion  of  the  concepts  behind  the  quality  of  verification 
approaches  and  the  oethodolcgy  used  tc  measure  that  quality  and  its 
associated  costs  was  described  in  earlier  sections. 

This  section  describes  the  application  of  that  methodclogy.  It  is 
presented  in  three  parts.  The  first  part  describes  the  raw  data 
gathered  and  the  techniques  used  tc  gather  that  data.  The  second  part 
analy7es  and  sumoarizes  this  data.  Finally,  the  third  part  presents 
an  interpretation  of  the  results. 


7.3.1  Data  Collection 


Tvic  distinct  steps  have  been  identified  as  necessary  for  estimating 
the  costs  to  be  assigned  to  the  different  verification  approaches. 
These  have  been  discussed  in  detail  previously.  They  are  summarized 
here : 

Step  1  Establish  the  relationship  between  the  number  of 
architectural  discrepancies  remaining  in  a  computer 
after  architectural  verification  (sell-off)  and  the 
cost  of  permitting  those  discrepancies  (because  of  an 
imperfect  verificaticn  method). 

Step  2  For  each  verificaticn  method,  determine  the  number  of 
architectural  discrepancies  expected  tc  remain  after 
verification,  and,  using  the  relationship  established 
in  Step  1,  compute  the  cost  associated  with  those 
remaining  architectural  discrepancies. 


7.3. 1.1  Data  Collection  for  Step  1 


Five  avicnics  programs  were  identified  in  the  proposal  to  the  Air 
Fcrce  as  potentially  useful  for  providing  data  for  Step  1.  These 
were : 

PAVE  LOW 
A-7 

B-52D  SPN/GEANS 
F-1  1  1 

Z-3A  AWACS 

In  the  course  of  study,  nine  other  potentially  useful  programs  have 
teen  identified.  Each  cf  these  is  a  hardware  upgrade.  (See  Section 
-.2.1.3  for  a  discussion  of  the  significance  of  the  hardware  upgrade 
aspect.)  These  programs  are: 
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P  V  ’■:  L  Ac'  K 
PA'.:  -  a  ;'/ VAT? 

F  -  : 

Spa^  -  '>. 

A-6  L 

A-6  J  '■'issile  Wiring  System 

lAHP? 

EA-  DC 
f-8 

All  ■  .en  pro  •  re;  investigated  tc  obtain  a  general 

v;ncle,.s  ng  of  ea  collected  included  the  mission,  customer, 

cctrrrt-  .  inve^vr  '  ••off  date,  nature  of  change,  and  the  names 

cf  t  ;  adce,  s-  :  v  «  i  ,  configuration  control,  and  Program  Office 

censor,  rei. 

These  programs  were  then  scrutinized  for  applicability  of  data.  Eight 
cf  the  fourteen  programs  were  eliminated.  Various  reasons  apply.  For 
example,  the  Space  Shuttle  software  validation  costs  were  not  deemed 
representative  ox  the  validation  costs  which  would  be  expected  for  the 
anticipated  KIL-STD-1750  applications  because  cf  the  special 
reliability  concerns  due  to  large  investments  in  the  program  and 
redundancy  testing  necessary  for  manned  safety.  As  ancther  example, 
the  fi-f  Univeesai  llissile  Wiring  System  was  only  in  the  final  checkout 
stage  at  the  time  of  data  collection.  Hence,  the  data  is  not 
available. 

The  avicriwcs  upgrade  programs  which  remain  are: 

E-52D  3PN/GERNS 

A  -  / 

PAVE  lACK 
PAVE  TACK/VATS 
F-n  1 
PAVE  LOW 

The  computers  involved  in  these  programs  are  shown  in  Table  7-14. 
retailed  software  validation  data  and  EC  data  were  then  collected  for 
these  programs.  These  data  are  listed  ir  Tables  7-15  through  7-20, 
respectively. 
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Table  7-14.  Harauare  Upgrades 


Sell-Off 

1  Original  Coinputer/I  New  Coaputer/ 

Ptcgram 

Date 

1  Fart  Number 

1  Part  Number 

A-7 

10/15/76 

1  TC-2A{74) 

1  TC-2A  (76) 

1  6870600 

1  6670500-1 

1  (Pre-Prod) 

1 

1  (Prod) 

1 

PAVE  LCW 

01/26/79 

1  TC-2A  (76) 

1  TC-3E 

1  6870500-2 

1  6259000-1 

rWE  TACK 

04/01/76 

1 

1  TC-3 

1 

1  TC-3 

1  687C400-1 

1  6870400-1 

PAVE  TACK/VATS 

02/28/78 

1 

1  TC-3 

I 

1  TC-SA 

1  6870400-1 

1  6870400-2 

1  (Pre-Fred) 

1 

1  (VAIS) 

1 

F-111 

02/17/78 

1  CP-2 

1  CP-2A 

1 

1  6195500-1 

E-52D 

SFN/GIANS 


2/24/79 


AP-101 

6160C25 


AP-101C 

6217800-20 


p 
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Table  7-15.  A-7  Program  Data 


+ - - 

1  Frcctam:  A-7  I 
i  '  I 
I  Customer:  Navy/LTV  I 
I  I 
I  Computers:  TC-2A  (74),  converted  to  TC-2A  (76)  | 


I  Sell-Off  Date:  15  October  1976 
I 

I  Operational  flight  Program  Validation:  Naval  Weapons  Center, 
I  China  Lake 


I  Function  Description:  Navigation  and  weapons  delivery  for  j 
(Close  Support  Light  Attack  aircraft.  1 

lEardware  Change  Summary:  Changed  from  Kodular  Core  Memory  (HCa)  I 
(to  Double  Density  Modular  Core  Meitcty  (LMCM)  ,  increasing  maximum  | 

I  memory  capacity  from  40K  by  16  bits  to  64K  by  17  hits.  Added  memory  | 
(parity,  hardened  base  registers,  and  five  instructions.  Optimized  ( 
(I/O  logic  and  increased  CFO  performance.  Added  surge  amp  in  converter ( 
(for  wheels  drive.  I 

(Software  fievalidation  Data:  The  navigation  and  weapons  delivery  | 
(program  was  revalidated  by  the  Navy  at  the  China  Lake  Naval  Weapons  ( 
(Center.  Ihe  program  was  16K  in  length.  Effort  expended  was  96  man-  ( 
(weeks.  The  initial  validation  effort  was  performed  or  a  laboratory  ( 
(simulator,  then  30  flight  tests  were  performed.  ( 

(EC  Data:  Twenty-four  architecture  related  changes  have  been  made  ( 
(since  sell-off.  (See  the  discussion  about  the  scope  of  ( 
("architectural  relevance"  in  the  Section  5-2. 1.2.)  These  ECs  are  ( 
(plotted  in  Figure  7-4.  ( 
> - -  -  -  -  -  -  -  -  + 
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Table  7-16.  3-52D  SFN/GEANS  Program  Data 


♦ - - - - - - - - - - - - - 

I  Prcgrao;  B-52D  SPN/GEANS  I 

I  I 

I  Custonei;:  Air  force/Harner  Ecbbins  Air  logistics  Certer  I 

I  I 

I  CcBDUters:  AP-101,  converted  to  AP-101C  I 

I  I 

I  Sell-Off  Date;  12  March  1979  I 

I  I 

1  Ccerationai  Flight  Program  Validation:  lEM  I 

I  I 

, - I 

(Function  Description;  The  computer  is  used  to  ccnticl  an  Inertial  1 

(Measuring  Unit  (platform)  for  precision  navigation  of  the  3-52D.  ( 

(This  unit  is  designated  SPN/GEANS  (Standard  Precision  Navigation/  I 

(Gimhalled  Electrostatically  Suspended  Gyro  Airborne  Navigation  1 

[  System)  .  I 

I  I 

( fiardvare  Change  Summary:  The  AP-101  computer  was  replaced  with  the  j 
(JP-IOIC.  This  involved  a  complete  re-packaging  of  the  macnine,  I 

(including  CPU,  I/O,  and  memory.  ! 

I  I 

(Software  Revaiidation  Data;  Software  revalidation  began  on  12  March  i 
(  1979.  Approximately  1  '3  of  the  effort  was  for  adding  new  functions;  | 

(the  data  has  this  taken  into  account.  Flight  testing  has  not  begun.  | 

(Therefore,  estimates  of  the  total  expected  number  of  fligut  tests  have  I 
(teen  made  by  personnel  experienced  in  flight  testing.  Projected  I 

(effort  is  43  man  weeks.  Approximately  twelve  flight  tests  are  I 

(expected.  The  size  of  the  program  being  revalidated  is  18K.  I 


(EC  Data;  Twenty-six  ECs  of  architectural  relevance  have  been  written 
[since  sell-off.  (See  the  discussion  about  the  scope  cf  "architectural 
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Table  7-17.  PAVE  TACK  Frogram  Data 


+  ____ - - - —  _ — - - - - - - - —  - - - - - - - - + 

I  Fccgram:  PAVE  TACK  | 

I  I 

I  Customer:  Air  Force/Ford  Aerosoace  | 

I  '  I 

1  Computers:  TC-3  1 

I  I 

1  Sell-Off  Date:  1  April  1976  | 

I  '  I 

1  Cperaticnal  Flight  Program  Validation:  IE?  ( 


[Function  Description:  The  functict  is  to  identify  targets  fcr  | 
[precision  electro-optical  weapons  using  a  laser  ranger/designator  pod. | 
[  A  TV  displays  target  information  tc  operator  and  a  pod  inferfaces  [ 
[with  the  NAV  computer.  This  is  used  on  both  the  F-4  and  E- 1 1 1  ( 
[platforms-  1 

I  I 
[Hardware  Change  Summary:  The  memory  was  doubled  in  size  from  8K  to  ) 
[16K,  by  converting  from  one  MCK  tc  two  ECM. 

I  I 
[Software  Eevalidation  Data:  Two  forms  cf  self-test  facilitated  j 
[checkout.  The  first  was  a  minimal  test  which  is  invoked  periodically  j 
[during  the  execution  of  the  Operational  Flight  Prcgram.  The  second  [ 
[was  a  Self  Test  aode  which  drives  the  pod  hardware  and  feeds  back  [ 
[data  for  a  closed-loop  check.  [ 
I  i 
[The  program  which  was  revalidated  was  8K  in  size  and  took  one  man  week| 
[plus  one  flight  test  to  revalidate.  i 

[ FC  Data:  One  EC  of  architectural  relevance  was  written  after  sell-off j 
I  (1  April  1976).  [ 
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Table  7-18.  PAVE  LON  Program  data 


+  - __________________ - - — _.f 

I  Program:  PAVE  LON  | 

I  I 

I  Customer:  Air  Force/ASD/SDX  | 

I  I 

I  Computers:  TC-2,  converted  to  TC-3B  | 

I  I 

I  Sell-Off  Date:  26  January  1979  | 

I  I 

I  Operational  Flight  Program  Validation:  lEK  | 


Function  Description:  The  PAVE  LON  mission  is  a  helicopter  search 
and  rescue  during  night/adverse  weather.  The  equipment  performs 
precision  navigation  and  locates  survivors.  The  computer  estimates 
survivor  latitude  and  longitude  (based  on  "beeper"  information)  and 
manages  a  platform,  IE  sensors,  and  terrain  following  radar. 

Hardware  Change  Summary:  The  unique  PAVE  LOW  I/O  was  incorporated 
into  the  computer  LED.  Processor  speed  was  increased,  hemcry  size 
increased  from  16K  to  32K  by  replacing  the  Easic  Operating  Memory 
with  DMCM.  New  instructions  were  added  and  the  logic  was  optimized. 

Software  Bevalidation  Data:  Eevalidation  effort  consisted  of  four 
nan  cays  of  lab  effort  and  one  flight  test.  The  program  which  was 
revalidated  was  16K  in  length. 

EC  Data;  Only  one  architecture  related  EC  was  written  since  sell-off 
on  26  January  1979. 
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Tacle  7-19.  PAVE  TACK/VAIE  Program  Data 


Program;  PAVE  TACK/VATS 
Customer:  Air  Force/Ford  Aerospace 
Computers;  TC-3 

Sell-Ofr  Date:  28  February  1978 
Ccerationai  Flight  Program  Validaticr:  TEK 


Function  Description:  The  function  is  to  identify  targets  for 
precision  electro-optical  weapons  using  laser  ranger/designator  pod. 

A  TV  displays  target  information  to  operator  and  a  pod  interfaces  with 
the  NAV  computer.  This  is  used  on  both  the  F-4  and  F-111  platforms. 

Hardware  Change  Summary:  Additional  I/C  channel  capability  was  added 
to  interface  with  the  Video  Augmented  Tracker  System.  This  allowed 
a  50  KH2  input  channel  to  be  multiplexed  between  the  laser  ranger  and 
the  Video  Tracking  Unit. 

Software  fievalidation  Data:  Two  forms  cf  self-test  facilitated 
checkout.  The  first  was  a  minimal  test  which  is  invoked  periodically 
during  the  execution  cf  the  Operational  Flight  Program.  The  second 
was  a  Self  Test  iiode  which  drives  the  pcd  hardware  and  feeds  back  data 
for  a  closed-loop  check. 

The  program  which  was  revalidated  was  12K  in  size  and  took  two  man- 
weeks  plus  one  flight  test  to  revalidate. 
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Table  7-20.  F-111  Prcgran  Data 


- - - - + 

1  Prcqram;  F-111  | 

I  I 

I  Customer:  Air  Force/SHALC  | 

I  I 

I  Computers:  CP-2,  converted  to  CP-2A  | 

I  I 

I  Architecture  Verification  Date:  17  February  1978  | 

I  I 

I  Cperational  Flight  Program  Validation:  Sacramento  Air  Logistics  | 

I  Center,  Ingineering  Division  | 


Function  Description:  The  F-111  is  a  tactical  f ighter-tcmter  for 
deep  interdiction  and  strategic  missicns.  The  computers  perform 
navigation  and  weapons  delivery. 

Eardware  Change  Summary:  The  CP-2  machine  was  completely  redesigned 
and  repackaged,  while  retaining  instructicr  set  compatibility. 

Storage  capacity  was  increased  from  16K  by  18  bits  to  tUK  by  18  b’ ts. 
Performance  was  increased  by  a  factor  of  three. 

Software  Eevalidation  Data:  The  CP-2A  brassboard  machine  was  sub¬ 
mitted  to  the  Air  Force  as  part  of  an  unsolicited  proposal.  This 
machine  was  evaluated  by  validation  of  proper  execution  of  various 
F-111  Operational  Flight  Programs.  Tests  were  conducted  at  the  F-111 
Avionics  Integration  Support  Facility,  using  both  static  and 
simulated  flight  conditions.  No  flight  tests  were  performed;  however, 
it  was  estimated  that  six  flight  tests  would  be  necessary.  Validation 
was  completed  for  three  OFPs  which  were  20K,  16K,  and  16K  in  length. 
The  20K  program  was  a  ccmposite  of  the  ether  two.  Effort  expended 
for  these  programs  was  54  man  weeks. 

IC  Data:  Since  the  machine  which  was  used  for  validation 
was  a  breadboard,  changes  were  recorded  but  not  formally  written  up 
as  ECs.  Hence,  no  time  information  is  available.  A  tctal  of  nine 
architectural  discrepancies  were  found.  Testing  took  place  over  a 
2  1/2  month  period.  Dse  of  the  machine  was  then  discentinued. 

(Note  that  this  machine  was  in  the  form  of  an  unsolicited  proposal; 
it  was  not  sold.) 
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IC  data  collection  typically  began  by  obtaining  an  "As-Euilt  List"  tor 
the  ccraputer  in  question.  This  list  contains  the  part  numbers  for  all 
of  the  major  subassemblies  in  each  computer.  Summary  EC  information 
was  obtained  for  each  major  subassembly  in  the  computet  via  tne  Owego 
Part  Number  Identification  User  System  (CEIUS) .  This  is  an  automated 
system  Mhich  allows  a  user,  via  a  keyboard/display  terminal,  to  obtain 
a  listing  of  all  ECs  written  against  each  part  number. 

Each  EC  was  physically  pulled  from  the  records  center  and  examined  to 
determine  its  architectural  relevance.  (See  the  discussion  about 
architectural  relevance  in  Section  5- 2.  1.2.)  Each  EC  contains 
narked-up  drawings,  wiring  lists,  parts  lists,  and  a  description  of 
the  nature  of  the  change.  Fcr  each  EC  judged  to  be  relevant,  the  EC 
number  and  date  were  recorded. 

The  software  revalidation  data  listed  in  Tables  7-15  through  7-20  is 
summarized  in  Table  7-21.  The  cost  figures,  both  in  terms  of  total 
cost  and  cost  per  K  lines  of  code,  are  also  shown  in  Table  7-21.  Two 
cost  elements  comprise  the  total.  The  first  is  the  weekly  wage  cost, 
J1,346,  which  is  multiplied  by  the  number  of  man  weeks  of  effort.  The 
second  is  the  average  cost  per  flight  test.  Two  data  points  are 
averaged  here:  the  current  average  cost  used  by  the  Navy  at  the  China 
lake  Naval  Weapons  Center  which  is  $2,000  and  the  anticipated  cost  of 
the  flight  tests  for  F-111,  which  was  J3,CC0.  The  resulting  $2,500 
average  is  multiplied  by  the  number  of  flight  tests. 


Table  "^-21.  Operational  Flight  Program  Revalidaticn  Data 


1  PAVF 

1 

LOW 

1 

1 

TC-3B 

1 

1  PAVF 

1 

TACK 

1 

J 

TC-3 

1 

IPAVE 

TACK/VATS 

i 

I 

1 

TC-3  A 

1 

IF-111 

1 

1 

CP-2A 

I 

IE-52D/SFN  GEANS 

I - - 


AP-1 01C 


1 *Includ€s  projections. 


+ 


Bevalidaticn  Data 


Program 

Size 

1 

1  MW 

(Flight 

1  Tests 

- 1 

Total  1 
Cost  1 

Cost  per 
K  Lines 
of  Code 

16K 

1  96 

1  30 

204,216) 

12,764 

16K 

1 

1  1 

1 

1  1 

I 

3,846  I 

240 

8K 

1 

1  1 

! 

I  1 

3,5461 

481 

12K 

1 

1  2 

I 

1  1 

5,  192) 

433 

20K,16K, 

16K 

1 

I  54 

1 

1  6* 

1 

1 

87,664) 

1 

1,686 

18K 

1  “O 

1 

1  12* 

87,678) 

4,882 

+ 
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7.3.  1.2  Bata  Collection  for  Step  2 


Ite  application  of  Step  2  described  above  requited  the  collection  of 
EC  data  for  three  types  of  verification  approaches: 

Functional 

Bandom 

Lockstep 

For  the  Functional  approach,  three  of  the  programs  which  have  already 
teen  investigated  are  applicable.  These  are  B-52D  SPN/GEAKS,  A-7,  and 
F-111.  The  machines  in  these  programs  underwent  complete,  or  nearly 
ccmplete,  changes.  As  a  result,  the  data  should  represent  the  results 
that  could  be  expected  for  application  of  a  Functional  type 

verification  process  to  a  typical  MIl-STD-1750  machine.  (Note  the 
deficiencies,  however,  in  the  F-111  data,  described  in  Table  7-20.) 
The  EC  data  in  Tables  7-15,  7-16  and  7-2C  apply. 

Fcr  the  Bandom  approach,  an  lEK  S/370  Model  was  investigated. 
Or fortunately,  no  machines  have  been  built  which  were  tested  solely 
with  the  Bandom  approach.  In  all  cases,  the  Functicnal  method  Wc^s 
also  acplied.  The  test  package  consisted  of  a  control  program 

(operating  system)  and  the  following  tests: 


-  Storage 

-  Relocation  Architecture 

-  Operating  System  Hardware 

-  Storage  Protection 

-  Miscellaneous  I/C 


Key  to  this  package  is  the  use  of  the  randcm  instruction  generator. 

Bata  gathered  were  Bequests  for  Engineering  Action  (EEAs)  which  are 
equivalent  to  Engineering  Changes.  Each  PEA  provides  a  description  of 
the  change  and  the  reason  for  the  change.  Each  FEA  was  examined  as  to 
its  relevancy;  non-architecture  related  changes  were  thrown  out. 

First  ship  was  March  17,  1978.  REAs  prior  to  this  date  were  ignored. 
Change  data  are  shown  in  Figure  7-6. 
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ECs 

found 


Months  after  sell-off 

Figure  7-6.  ECs  of  Architectural  Eelevance  for  Syst6n’/370  Model 

versus  Tire 


For  the  Lockstep  approach,  the  IBM  Series/1  Model  4S52  was  chosen. 
This  method  has  been  employed  by  the  lEK  General  Systems  Civisicn  to 
test  several  models  of  the  Series/1  ccmputer:  the  4953,  4955,  and  the 
4S52.  Lockstep  had  been  applied  to  the  first  two  models  subsequent  to 
first  ship.  Only  the  Model  4952  had  teen  verified  by  the  Lockstep 
irethcd  prior  to  first  customer  ship.  First  ship  date  was  June  4, 
1979.  At  the  time  of  data  collection,  August  29,  1979,  no  Engineering 
Changes  had  been  written  against  this  model. 

Also  required  for  the  application  of  Step  2  was  the  collection  of 
machine  size  and  EC  distribution  information  for  the  rormalization  of 
the  EC  data.  The  methods  for  application  of  these  data  are  described 
in  Appendix  B. 

The  machine  size  data  was  gathered  with  the  use  of  the  OPIUS  system 
and  computer-generated  indented  parts  lists.  For  upgraded  computers, 
each  subassembly  parts  list  was  compared  for  differences  between  the 
original  and  the  upgraded  machines.  These  differences  were  tabulated 
i  and  totaled  to  arrive  at  the  machine  change  size  in  terms  of  gates, 

microcode  bits,  and  main  storage  bits.  These  data  are  shown  in  Table 
7-22. 
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Table  7-22.  Hardware  Change  Size/Machine  Size 


Program 

1  Logic 

1  (Gates) 

1  Microcode 

1  (Bits) 

1  Main  Store 

1  (Bits) 

A-7 

1  15,350 

1  0 

1  458,  752 

PAVE  LCH 

1  7,519 

1 

1  0 

1 

1  0 

PAVE  TACK 

1 

1  0 

1 

1  0 

1 

1 

1  131,072 

1 

PAVE  TACK/VATS 

1 

i  ^ 

1 

1  0 

1 

1  0 

F-11  1 

1  15,715 

1 

1  547,136 

1 

1  1,179,648 

E-52D/SPN  GEAKS 

1 

1  29,894 

1 

1  611,201 

1 

1 

1  1,179,648 

1 

Seri es/1 ♦ 

1 

1 

1 

System/370  Model 

1  274,000 

I  1,719,296 

1 

1  8,368,608 

I  *  Since  no  ECs  were  fcund,  no  machine  size  data  collection  was  1 

I  necessary.  1 

+  - - — - - - - - - - - -  - - - 

Each  relevant  EC  was  ezaminefi  to  determine  whether  the  cause  was 
logic-related,  microcode-related  or  mair  store-related.  These  data 
are  shown  in  Table  7-23. 
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Table  7-23.  Distributior  of  FCs 


Program 

1  logic 

I  Kicrocode 

1  ilain  Store 

A-7 

1  16 

I  0 

1  3 

PAVE  LOH 

1 

1  0 

I 

I  0 

1 

1  1 

PAVE  TACK 

1 

1  1 

I 

I  c 

1 

1  0 

PAVE  TACK/VATS 

1 

1  c 

1 

I 

I  c 

1 

1 

1  0 

F-11  1 

1 

1  6 

1 

1  1 

1 

1  2 

E-52E/SPN  GEANS 

1 

1  16 

I  10 

1 

1  0 

Series/1* 

I 

1 

1 

System/370  Model 

I  26 

1  lh 

1 

1  ^ 

♦Since  nc  ECs  were  found,  no  machine  s 
necessary. 

ize  data  collection  was 

The  procedure  for  normalization  of  architectural  discrepancies 
requires  information  on  the  size  of  the  ncninal  MIl-STE- 1 7,50  machine. 
Tie  ncfflinal  size  of  the  BIL-STD-1750  machine  is  assumed  to  be  the  same 
as  the  machine  used  in  E-52D  SES/GEANS.  This  machine,  which  was 
ecently  developed,  contained  64K  by  32  bits  cf  main  storage,  two 
IL-STD-1553  channels,  and  discrete  I/O.  Performance  is  approximately 
CO  KCPS.  The  unit  is  packaged  in  a  full  ATE  configuration.  This  is 
elieved  to  be  typical  of  the  forthcoming  MIl-STE-1750  machine. 


7.3.2  Analysis  of  Quality  Data 

This  section  contains  an  analysis  of  the  data  descriled  in  the 
previous  section.  It  begins  with  the  estimaticn  of  the  total  number 
of  architectural  discrepancies  for  the  programs  previously  described. 
This  is  done  with  the  aid  of  a  set  cf  APL  programs  which  were  written 
to  implement  the  correlation  coefficient  and  squared  error  techniques 
described  in  Appendices  E  and  C. 

Next,  the  relationship  between  the  projected  total  number  of 
architectural  discrepancies  remaining  after  verification  and  the  cost 
of  having  those  discrepancies  remain  is  established. 

Finally,  the  quality  associated  with  each  verification  method  is 
discussed. 
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7.3.2. 1  Application  of  Correlation  Coefficient  and  Squared  Error 
Techniques  tc  EC  Data 


Engineering  Change  data  for  the  various  programs  ard  machines  has 
previously  been  described.  To  Eumiiari2e  the  results,  sufficient  data 
exist  to  estimate  the  total  number  of  architectural  discrepancies  for 
three  machines/ptograms;  A-7,  B-52D  SEN/GEANS,  and  the  System/370 
Model.  Series/1,  PAVE  LOW,  PAVE  TACK/VATS,  and  PAVE  TACK  have  either 
zero  or  one  EC;  no  projection  is  therefore  possible.  The  F- 1 1 1  data 
has  no  time  information  and  the  method  cannot  be  applied. 

To  aid  in  determining  the  projected  total  number  of  architectural 
discrepancies  for  A-7,  B-52D  SPN/GEANS,  and  the  Systen/370  Model,  APL 
programs  have  teen  written  which  apply  the  ccrrelatioh  coefficient  and 
squared  error  techniques  which  are  descrited  in  Appendix  B-  Five 
programs  have  been  written: 


X  FIT  Y 
F 

MEAN 

VAE 

ESTIMATE 

These  programs  are  listed  in  Appendix  C.  The  prccram  X  FIT  Y 
transforms  the  cumulative  data,  g  (t) ,  and  computes  the  best  fitting 
line  using  the  linear  regression  formulas  given  in  Appendix  B.  It 
calls  upon  the  programs  MEAN  and  VAF  tc  compute  the  means  and 
variances  required  for  the  regression  analysis.  X  FIT  Y  also  computes 
the  correlation  coefficient  and  the  squared  error.  For  the  latter,  it 
calls  upon  the  program  F  to  calculate  the  values  of  the  fitted  curve, 
9  It  )  • 

Finally,  the  program  ESTIMATE  is  used  tc  estimate  the  total  number  of 
architectural  discrepancies.  It  does  sc  by  successive  approximation. 
It  chooses  values  of  C  (as  previously  discussed)  and  invokes  X  FIT  Y 
tc  provide  the  corresponding  correlation  coefficients  and  squared 
errors.  It  determines  how  to  change  C  tc  obtain  either  a  higher 

correlation  coefficient  or  a  lower  s  red  error.  It  continues  to 

test  different  values  of  C  until  th-  b  estimate  of  C  is  obtained. 

Notice  that  the  program  ESTIMATE  ,.s  listed  in  Appendix  C  prints  "A-7 
Ergineering  Changes".  This  is  changed  fcr  each  new  set  cf  data. 

Also  note  that,  as  listed,  it  optimizes  based  on  the  sguared  error.  A 
simple' change  maxes  it  optimize  on  the  ccrrelaticn  coefficient. 

The  souared  error  is  a  better  measure  cf  fit,  since  it  compares  the 

calculated  curve,  g(x),  with  the  actual  data.  The  correlation 

coefficient,  however,  measures  the  degree  cf  fit  of  the  calculated 
line  with  the  transformed  data.  Since  the  transform  is  non-linear, 
error  may  be  introduced.  Because  of  this,  the  sguared  error  technique 
is  used  for  estimation  of  C.  However,  the  calculations  are  also 


7-66 


6 176175A 


FINAL  FEPCPT 


Fekruary  2y,  19fc0 


performed  using  the  correlation  coefficient  technique,  since  it  is 
expected  that  the  errors  introduced  would  te  small.  He  results  serve 
as  a  check  of  the  squared  error  method,  and  are  provided  for 
information  purposes  only. 

The  results  of  these  programs  are  provided  in  Appendix  D.  They  are 
summarized  in  Table  7-24. 


Table  7-24.  Estimate  of  Total  Architectural  Discrepancies 


- - - 

I  I  I  Estimated  Total  Architectural  Discrepancies  | 

I  I  Total  I - j 

I  Program  I  ECs  l  Squared  Error  |  Correlation  Coefficient  | 

I  I  I  Method  I  Method  I 

I  E-f2D  SEN/  )  26  I  27  |  30  I 

I  GIANS  III  I 

III  I  I 

I  A-7  I  24  I  29  I  26  | 

III  I  I 

I  Systeffl/370  III  I 

I  Model  J  44  I  68  |  66  i 


It  is  satisfying  to  note  that  the  projections  using  the  two  different 
methods  track  quite  .  closely.  it  is  also  satisfying  tc  ncte  that  the 
projections  for  the  System/370  Model  using  the  two  methods  compare 
favorably  with  the  projection  of  64  total  architectural  discrepancies 
that  was  obtained  previously  using  the  decreasing  exponential  method. 


7. 3. 2. 2  Projections  for  Programs  with  Insufficient  Data 


As  was  mentioned  previously,  five  of  the  programs  dc  not  have 
sufficient  data  to  permit  use  of  the  foregoing  methods.  Four  of  these 
five  have  either  zero  or  one  EC  written  against  them.  Since 
insufficient  data  points  exist  to  fit  lines  for  these  four  programs, 
the  data  will  be  used  as  is;  i-e.,  the  projected  tetai  number  of 
architectural  discrepancies  is  the  same  as  the  number  of  ECs  found  to 
date . 

The  fifth  program  is  F-111,  which  has  nine  architectural  discrepancies 
tut  DC  time  informaticn.  Because  of  this,  the  methods  of  projection 
of  the  total  number  of  architectural  discrepancies  carnet  be  applied. 
Again  the  data  is  used  as  is;  nine  architectural  discrepancies  is  the 
crojected  total  for  F-111.  This  necessarily  biases  the  data,  but 
there  is  another,  offsetting,  bias.  This  will  be  disccssed  more  fully 
in  Section  7.3. 2.4. 
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7. 3. 2. 3  Estimation  of  the  Cost  vs  Architectural  Discrepancies 
Pelationship 


Ite  cost  of  validation  of  Operational 
the  projected  number  of  architectural 
verification  is  plotted  in  Figure  7-7. 
ir  Table  7-25.  These  data  have  been  pre 


Flight  Programs  (CFPs)  versus 
discrepancies  remaining  after 
The  data  are  also  summarized 
viously  described  in  detail. 


THOUSANDS  OF 
DOLLARS  PER  K 
LINES  OF  CODE 


Figure  7-7.  Cost  versus  Architectural  Discrepancies 
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""able  7-25.  Summary  of  Architectural  Ciscrepancy  and  Fevaiidation 

Cost  Data 


Total  Frcjected  |  Total  fie validation 
Architectural  |  Cost  Fer  K  Lines 


Program 

1  Computer 

1 

Discrepancies 

1 

of  Code 

A-7 

1  TC-2A 

1 

1 

29 

1 

1 

12,764 

FAVF  LOW 

j  T  C  ^ 

1 

1 

1 

1 

1 

24  0 

FAVF  TACK 

1  TC-2 

1 

1 

1 

1 

1 

4  8  1 

PAVE  TACK/VATS 

1  TC-3A 

1 

1 

C 

1 

1 

433 

F-  1 1  1 

1  CP-2A 

1 

1 

9 

1 

1 

1,666 

E-52D/SEN  GEANS 

1  AP-101C 

1 

1 

27 

1 

1 

4,682 

The  test  fitting  line  is  alsc  drawn  in  Figure  7-7.  The  equation  for 
the  line  was  calculated  via  linear  regression  analysis  using  the 
eauations  for  slope  and  intercept  described  in  Appendii  E.  The 
equation  is: 


Dollars 


K  Lines  of  Code 


154  +  320  * 


Projected  Total 

Number  of  Architectural 

Discrepancies 


(7-1) 


The  correlation  coefficient  for  this  equation  is  0.£77.  This  is  a 
measure  of  the  degree  of  fit  of  the  given  points  to  the  least  squares 
straight  line.  A  value  of  one  indicates  exact  correlation;  a  value  of 
zero  indicates  no  correlation.  In  this  case,  a  reasonable  fit  of  the 
data  to  the  least-squares  straight  line  is  indicated. 

The  negative  value  of  the  intercept  warrants  discussion.  If  a  near 
perfect  verification  program  were  available  (i.e.,  the  projected  total 
number  of  architectural  discrepancies  after  verification  would 
approach  zero),  the  minimum  cost  to  be  incurred  would  be  the  cost  of 
the  basic  flight  test  to  validate  the  operational  flight  program,  or 
S2,500.  To  translate  this  to  dollars  per  K  lines  of  code,  this  figure 
could  be  divided  by  the  average  size  of  the  Operational  Flight 
Programs,  in  this  case  15K.  This  yields  a  value  (which  is  tne 
expected  y-intercept)  of  S167  per  K  lines  of  code. 

tiaturally,  a  negative  cost  for  this  flight  test  is  physically 
impossible.  However,  when  the  least-squares  line  of  Figure  7-7  is 
examined,  it  can  be  easily  seen  that  a  y-intercept  of  J167  instead  of 
a  -$154  would  not  significantly  alter  the  fit  of  the  line  to  the  data 
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feints.  In  summary,  the  negative  intercept  does  not  severely  limit 
t^e  credibility  of  the  fitted  line. 

Note  that  this  curve  does  not,  by  itself  distinguish  betveen  different 
verification  methods.  But,  given  the  quality  (in  projected  total 
rumbei  of  architectural  discrepancies)  cf  a  particular  certification 
method,  the  associated  cost  can  be  determined. 

As  an  example  of  the  use  of  this  equation,  suppose  that  Verification 
Method  A  were  determined  to  have  20  expected  architectural 
discrepancies  remaining  in  a  machine  the  size  of  a  typical 
MII-STD-1750  implementation  after  certification.  This  would  yield 


Dolla  rs 


K  Lines  cf  Code 


-Ifh  +  320  ♦  20  (projected  architectural 

d iscrepancies) 


6,246 


If  one  assumed  an  average  Operaticnal  Flight  Program  size  cf  32K  and 
expected  the  numner  of  Operational  Flight  Programs  to  he  written  over 
the  useful  life  of  MIL-STD- 1 750  to  he  ten,  then  the  total  cost  is; 


cost  =  6246  Dollars  32  K  Lines  of  Code 

K  Lines  cf  Code  Operational  Flight  Program 

10  Operational  Flight  Programs 
=  $1,998,720 


This  may  be  tnought  of  as  an  estimate  cf  the  penalty  (cost)  that  must 
he  paid  for  having  a  less-than-perfect  verification  metnod 
(hypothetical  Verificaticn  Method  A,  in  this  example). 


7.3. 2. 4  Discussion  of  the  Data  Feints  for  the  Cost  versus 
Architectural  Discrepancies  Felationship 

Cf  the  six  data  points  used  in  ccnstructing  the  cost  versus 
architectural  discrepancies  relationship,  two  have  restrictions  which 
reduce  the  confidence  which  can  be  placed  in  them. 

The  first  is  B-52D  SFN/GEAN£.  This  program  has  yet  to  begin  the 
flight  test  portion  of  software  validation.  Hence,  an  estimate  was 
made  of  the  expected  num.ber  of  flight  tests  by  a  software  engineer 
with  significant  experience  in  this  area.  Naturally,  the  data  point 
is  no  better  than  the  estimate.  Vhile  cemparisor  with,  the  A-7  data 
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point  suggests  that  the  estimated  number  of  flight  tests  may  be  low 
and  that  the  cost  for  S-f2D  SPN/GEANS  may  be  understated,  the  data 
point  is  used  as  is  with  the  above  caveat. 

The  second  data  point  in  F-111.  Both  the  software  validation  cost  and 
the  projected  number  of  architectural  discrepancies  are  known  to  be 
understated.  The  reason  for  the  cost  bias  is  that  one  of  the  programs 
which  was  validated  was  a  composite  of  the  two  others.  As  a  result, 
some  of  the  code  which  was  validated  was  redundant;  the  cost  data  is 
an  understatement  of  the  true  cost. 

The  reason  for  the  bias  in  the  projected  number  of  architectural 

discrepancies  lies  behind  the  incomplete  nature  of  the  program.  IBM 

submitted  an  unsolicited  proposal  to  replace  the  twc  CP-2  computers 
onboard  the  f-111  with  a  CP-2A  computer.  A  brassboard  machine  was 
built  and  was  used  for  Operational  Flight  Program  revalidation  by  the 
Air  Fcrce.  Only  the  simulation  portion  of  validation  was  completed 

however;  no  flight  tests  were  performed.  Because  of  the  informal 

nature  of  the  brassboard  development,  changes  were  recorded  but  not 
formally  written  up  as  ECs.  Open  completion  of  the  simulation  portion 
of  the  validation,  use  of  the  machine  was  discontinued.  Estimates  of 
the  expected  number  of  flight  test  and  associated  costs  were  made  by 
the  Air  Force,  however.  In  summary,  the  following  deficiencies  exist 
with  the  F-111  data: 

1.  The  lack  of  time  information  does  not  permit  a  projection  of 
the  total  number  of  architectural  discrepancies.  Also,  the 
amount  of  data  has  been  reduced  because  use  cf  the  machine 
was  discontinued.  Since  the  number  of  ECs  is  used  directly 
as  an  estimate  of  the  projected  total  number  of 
architectural  discrepancies,  this  results  in  an 

understatement  of  the  total  number  of  architectural 

discrepancies. 

Since,  in  this  case,  the  number  of  ECs  is  used  directly  as 
an  estimate  of  the  projected  total  number  of  architectural 
discrepancies,,  this  results  in  an  understatement  of  the 
total  number  of  architectural  discrepancies. 

7.  The  flight  testing  portion  of  the  software  validation  costs 
is  an  estimate. 

The  three  OFPs  were  somewhat  redundant  in  nature,  resulting 
in  an  understatement  of  the  Validation  cost  per  K  lines  of 
code. 

Since  the  biases  for  the  F-111  data  point  are  opposing,  it  is  not 
clear  which  would  dominate  any  changes  tc  the  final  relationship  shown 
in  Figure  7-7. 

In  summary,  both  the  F-111  and  E-52  SPN/GEANS  data  points  are  taken  as 
is  with  the  caveats  about  credibility  abeve. 
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7. 3. 2. 5  Analysis  of  Quality  Data  for  Different  Verification  Methods 


As  suirnarized  in  a  previous  section.  Step  2  in  estimating  the  costs  to 
te  assigned  to  the  different  verification  approaches  requires  that, 
fcr  each  verification  approach,  the  number  of  architectural 
discrepancies  expected  to  remain  after  verification  he  estimated. 
This  wculd  then  be  applied  to  the  cost  versus  architectural 
discrepancies  relationship  obtained  in  the  previous  section. 

The  data  which  were  collected  for  Step  2  appeared  previously  in  Tables 
7-15,  7-16,  and  7-20  and  in  Section  7.3. 1.2.  They  are  summarized  in 
Table  7-26. 

Table  7-26.  Architectural  Discrepancies  Data  for  Different 

Verification  Methods 


Verification 

Method 

1  1 

1  Computer/Program  | 

1  1 

ICs  Found 

1 

1 

1 

Total  Projected 
Architectural 
Discrepancies 

Functional 

Type 

1  TC-2A/A-7  1 

1  1 

I  t 

24 

1 

1 

29 

1  1 

1  CP-2R/F-111  I 

1  1 

9 

i 

1 

1 

9 

1  AP-101C/  1 

1  B-52D  SPN/GFANS  | 

1  1 

26 

1 

1 

1 

1 

27 

FandcB  Type 

1  1 

1  Systeir/370  I 

1  Model  1 

44 

1 

1 

1 

6  8 

lockstep  Step 

1  1 

1  Series/I  I 

0 

1 

1 

0 

+ 


An  examination  of  Table  7-26  indicates  that  there  is  insufficient  data 
tc  satisfactorily  discriminate  between  the  three  verification  methods. 
While  the  data  for  the  Functional  type  certification  method  appear  to 
he  satisfactory,  the  single  data  points  of  the  Fandom  type  and 
Lcckstep  type  verification  methods  dc  not  provide  sufficient 
ccnfidence  to  permit  their  application  to  the  cost  model. 
Pirthermore,  the  Series/1  data  imply  that  the  Lockstep  type  method  is 
cf  perfect  quality.  This  contradicts  intuition. 

Because  cf  these  deficiencies  in  the  data,  there  is  not  sufficient 
ccnfidence  to  permit  distinction  between  the  different  verification 
rrsthods  on  the  basis  of  quality.  The  estimation  of  the  costs  to  be 
assigned  (Step  2)  will  not  be  fully  ccmpleted  and  the  normalization 
methcdolcgy  described  previously  will  not  be  fully  applied.  This  does 
net  completely  invalidate  the  results  cf  the  quality  investigation, 
however.  Seme  analysis  can  be  performed  with  the  recognition  that 
crly  limited  ccnfidence  can  be  obtained  in  the  results  and  the  results 


7-92 


C" 


6176n5^  FINAL  PEPCFT  February  29,  1980 


mvst  be  affiled  with  caution.  This  analysis  appears  below.  The 
ireaning  of  the  inability  to  complete  Step  2,  as  well  as-  the 
conclusions  to  be  drawn  from  the  data,  are  discussed  in  Section  7.5. 
A  discussion  of  a  priori  expectations  regarding  quality  appears  in 
Section  7.4.6. 

Even  though  it  is  not  possible  to  distinguish  between  the  different 
verification  methods,  it  is  possible  to  obtain  a  rough  estimate  of  the 
cost  penalty  associated  with  the  use  of  the  Functional  approach.  Cf 
tte  three  Functional  data  points  in  Table  7-26,  the  F-111  data  point 
is  eliminated  (for  the  reasons  discussed  in  the  previous  section)  . 
The  projected  total  number  of  architectural  discrepancies  is  taken  to 
be  the  average  of  the  B-52  SFN/GEANS  and  A-7  data  points  (i.e.,  28). 
Since  these  two  data  points  are  for  machines  which,  are  roughly  the 
size  of  the  nominal  dlL- STC- 1 750  machine,  and  since  great  precision  is 
not  expected  for  only  two  data  points,  nor lalizaticn  by  machine  size 
(as  described  in  Appendix  B)  is  not  performed. 

Applying  28  projected  architectural  discrepancies  to  Equation  (7-1) 
yields; 


Collars 


K  Lines  of  Code 


-154  +  (320  «  29) 
3,  806 


This  cost  per  K  lines  of  code  is  then  multiplied  by  the  expected 
number  cf  lines  of  operational  code  to  be  written  over  the  lifetime  of 
EIL-STD-1750.  The  Air  Force  has  estimated  this  to  be  between  313. 2K 
and  522K  lines  of  operational  code.  The  expected  cost  range  is: 


313. 2K  lines  of  code  *  8,806  dollars/K  lines  of  code  =  $2,758,039 
522K  lines  of  code  *  8,806  dcllars/K  lines  cf  code  =  $4,596,732 


Therefore,  the  expected  cost  penalty  that  wculd  be  paid  for  having  a 
less-than- perfect  verification  method  (in  this  case,  the  Functional 
type)  is  between  $2.8  and  $4.6  million. 

Further  analysis  can  also  be  applied  to  the  System/370  model  data 
point.  Pecall,  that  this  data  point  is  contaminated  in  that  both  the 
Fandom  and  Functional  approaches  were  applied.  Therefore,  one  expects 
that  the  resulting  quality  would  be  higher  than  that  cf  the  Functional 
approach  alone. 

This  is  shown  to  be  the  case  after  the  data  is  normalized  by  machine 
size.  Table  7-23  shows  the  distribution  of  logic-related  ECs, 
n icr ccode- rela ted  ECs,  and  main  storage-related  ECs.  Table  7-26  shows 
that  44  ECs  were  found  and  that  the  projected  total  number  of 
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architectural  discrepancies  is  68.  The  projected  number  of 
Icqic-related  architectural  discrepancies  becomes: 


Projected  Logic-Belated  =  (68/^4)  ♦  (26  logic-related 

Discrepancies  ECs  found) 

=  tiO 


Ibe  corresponding  projected  number  of  microcode-related  and  main 
storage-related  architectural  discrepancies  are  22  and  6, 
respectively. 

Normalizing  by  machine  size  (using  the  data  from  Table  7-26  and  the 
rcninal  machine  size  for  KIL-STD- 17 50) ,  and  using  the  method  in 
fippendix  B,  one  obtains: 


Project  Number  of  29,89U 

Architectural  =  40  ♦  -  + 

Discrepancies  274,000 

611,201 

22  ♦  -  + 

1  ,719,296 

1,179,648 

6  »  - 

8,388,608 

=  13 

This  number  suggests  that  the  improvements  to  be  obtained  by  adding 
the  Eandcm  approach  to  the  Functional  approach  would  be  to  decrease 
the  expected  number  of  architectural  discrepancies  from  28  to  13. 

The  application  of  these  results  appears  in  Section  7.5. 
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7.4  CCMFAEISON 


In  this  section  all  twelve  approaches  are  compared  against  the  same 
five  guidelines  used  to  evaluate  each  approach,  Non-Recurring  Start-Up 
Costs,  Recurring  Costs,  Time  Required  to  Perform  Validation,  and 
Impact  to  SEAFAC  Resources,  Tables  7-27  and  7-28  summarize  the  data 
described  previously.  Ncte  that  the  total  size  is  the  summation  of 
tte  verification  program  and  whatever  other  control  programs  or  test 
cases  that  are  developed  to  support  the  verification  process. 
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Table  7-27.  CoitpariEcr  Suminary 


1 

1  Approach 

1  Total 

1  Si2€ 

Verification  | 

Time  1 

Total  Cost  over  10 
Years  (K  Dollars) 

I 

1  AN/AYK-ISA  ATP 

1 

J 

1  Manual 

1  96K 

48 

min 

1 

462.4 

1 

1  Master/Slave 

i 

1  104K 

11 

mini 

i 

1 

617 

1 

1 

21 

min2 

1 

1 

1  Fandom 

1 

1 

1 

1 

1  Manual 

1  162K 

86 

hrs. 

50  min  | 

962. 8- 

1 

1 

1 

682-8— 

1  Master/Slave 

1  170K 

6 

hrs. 

12  mini  | 

1  133-8- 

1 

1 

7 

hrs. 

15  min2  | 

853.  8— 

1 

1  RVP 

i 

1 

1 

1 

1  Manual 

1  2M 

15 

hrs. 

2  min  | 

1 210. 4 

1 

1  Master/Slave 

1 

1  2B 

13 

mini 

1 

1 

1397.4 

1 

1 

1 

1 

22 

min2 

1 

I 

1  Diagnostic 

4 

1 

i 

1 

1  Manual 

1  96F 

48 

min 

1 

735.4 

1 

1  Master/Slave 

1 

(  10hK 

12 

min* 

1 

1 

890.0 

1 

I 

22 

min2 

1 

1  FTP 

I 

i 

1  Manual 

I  96K 

48 

min 

1 

462.4* 

1 

I 

1 

1C85.4** 

1  Master/Slave 

j  1 04K 

10 

mini 

1 

617.0* 

I 

1 

11 

min2 

1 

1240.0** 

1 

1  lockstep 

J 

1 

1 

1 

1  Manual 

1  1 .  1M 

7 

hrs. 

40  min  1 

647.0 

1 

1 

1 

1 

875.0*** 

I 

1  Master/Slave 

1 

1  64K  + 

17 

mini 

1 

1 

1650.  1  + 

1 

1  64K++ 

30 

min2 

1 

1245. 4++ 

1 

1  1  .  1M 

1 

997. 1+++ 

*  Modify  Existing  FTP  -  Write  Nev  Siculator 

**  Write  New  FTP  --  Modify  Existing  Simulator 

***  Witt  Software  Trace 

+  With  "Golden”  Computer 

++  With  Simulator 

+++  With  Predetermined  Fesults 

1  Based  on  MIL- STD- 1553  Transfer  Fate  of  30K  words/second 

2  Based  on  BS  232  Transfer  Bate  of  3F  words/seccnd 
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Table  7-28.  Cost  Ccuparison 


Approach 

NfiSO 
(K  $) 

Recurring 

Cost/Com¬ 

puter 

(K  Dcllars) 

1  Total  1 

1  Recurring  | 

1  Cost  Over  | 

1  10  Years  | 

1  (F  Dollars) 1 

Total  Cost  Over 

10  Years 
(K  Dollars) 

SN/AYK-15A  ATP 

1 

1 

Cfanual 

259 

6.78 

1  203.4 

1 

462.4 

Master/Slave 

39  2 

7.5 

1  225.0 

617.  0 

Random 

1 

1 

Manual 

799- 

5.46 

1  163.8 

1 

962.8- 

519— 

1 

1 

682.8 — 

Master/Slave 

952- 

6.06 

1 

1  181.8 

1  133.8- 

672— 

I 

1 

853.  8— 

A7P 

1 

1 

Manual 

833 

12-58 

1  377.4 

1 

1210.4 

Master/Slave 

966 

14.38 

1 

1  a3i.4 

1 

1 

1397.4 

Eiagnostic 

1 

1 

Manual 

532 

6.78 

1  203.4 

735.4 

Master/Slave 

66  5 

7.5 

I  225.0 

1 

1 

890.0 

FTP 

1 

1 

1 

1 

Manual 

259* 

6.78 

1  203.4 

1 

462.4* 

882** 

1 

1 

1085. 4** 

Master/Slave 

392* 

7.5 

1 

I  225.0 

1 

1 

617.0* 

1015** 

1 

1 

1240. C** 

lockstep 

1 

1 

1 

1 

Manual 

448 

13.30 

1  399.0 

1 

847.0 

476*** 

1 

1 

875.0*** 

Master/Slave 

1207  + 

14.77 

1 

1  443.  1 

1 

1 

1650.  1  + 

943  +  + 

10.08 

1  302.4 

1 

1245. 4++ 

602+++ 

13.  17 

1  395.  1 

1 

I 

1 

1 

I 

997. 1+++ 

NOTES:  REFER  TO  TABLE  7- 

•27. 
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7 . t» .  1  Test  CoEfiqurations 


A  given  verification  approach  car.  eliminate  certain  lest 
Ccnf igurations  from  consideration.  A  liniting  factor  surrounding  the 
lanual  test  configuration  is  the  size  of  the  verification  program, 
because  under  the  manual  test  configuration,  program  loading  and 
starting  reguires  manual  intervention.  Therefore,  the  larger  the  size 
of  the  verification  program,  the  more  separate  leads  it  reguires.  The 
following  verification  approaches  can  be  ruled  out  because  of  having  a 
greater  than  practical  amount  of  manual  intervention; 


Sect  ion  Arproacb 

7.3.1  AVP  in  a  Manual  Test  Ccnf iguratior 

7.3.5  Lockstep  in  a  Manual  Test  Configuration 

7.3.7  Random  Instruction  in  a  Manual  Test  Configuration 

This  leaves  verification  programs  developed  based  on  FTP,  Diagnostic, 
and  AN/AYK-15A  approaches  which  are  feasible  under  a  manual  test 
configuration,  and  which  are  very  similar  in  design  and  function.  The 
major  advantage  (to  the  vendor)  of  a  verification  program  running 
urder  a  Manual  Test  Configuration  is  that  it  easily  allows  the  vendor 
to  "pre-test"  the  computer  before  bringing  his  unit  in  for 
certification  (the  source  code  for  the  verification  program  will  be 
made  available  to  him  prior  to  certif icaticr.) . 

All  approaches  are  feasible  to  operate  under  the  Master/Slave  Test 
Configuration.  The  disadvantages  though  is  some  difficulty  (to  the 
vendor)  of  "pre-testing"  when  the  vendor  doesn't  have  the  Easter/Slave 
configuration,  including  the  I/O  interface. 


7 . h .  2  WcE-hecurrinq  S t a rt -0 j  Costs 


The  difference  between  the  lowest  Mcn-Fecurring  Start-Up  cost  for 
developing  a  verification  program  (AM/AYK-15A  or  FTP  approaches  in  a 
Manual  Test  Configuration  at  $259,C(jO)  and  the  most  expensive  approach 
(Lockstep  in  a  Master/Slave  environment  with  a  "golden"  computer  at 
$1,207,000)  is  $948,000.  The  key  issue  surrounding  the  cost  of  the 
FTP  approach  in  a  Manual  Test  Configuration  is  that  the  3.7  man  year 
figure  is  based  on  modifying  an  existing  FTP.  Currently,  there  does 
ret  exist  an  FTP  for  the  MlL-STD-nsO,  but  one  will  exist  in  the  July, 
1S80  time  frame.  The  AN/AYK-15A  ATP  currently  exists  with 
documentation  supporting  its  design  and  use. 

A  second  factor  to  be  considered  is  the  line  item  for  KIl-STD-1750 
simulators  included  in  the  Lockstep  and  Fandom  Instruction  approaches. 
If  an  existing  MIL-STD-1750  simulator  could  be  installed  and  modified 
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tc  cun  in  conjunction  with  the  vecificaticn  program,  then  the  6  man 
years  development  figure  could  be  cut  tc  a  2  man  year  modification 
cost . 

Certain  recurring  costs  are  the  same  fcr  all  approaches.  They  are  the 
hcctstrap  and  control  programs  for  a  Master/Slave  Test  Configuration, 
and  the  source  tape  generation  program  (vhich  may  be  just  a  system 
utility  for  the  Manual  Test  Configuration) -  The  cost  cf  the  Master 
computer,  and  MIL-STD-1750  support  software  are  considered  to  he  zero. 


■/ .  u .  3  Fecurrinq  Cost 


Fecurring  costs  are  directly  propcrtional  to  the  time  required  to 
ccmplete  the  verification  process.  Therefore,  from  Tatle  7-27,  it  is 
readily  observable  that  only  one  approach  distinguishes  itself 
significantly  enough  to  be  ruled  unfeasible  for  implementation.  The 
Fandom  Instruction  in  a  Manual  Test  Configuration  requires  11  work 
cays  of  manual  intervention  to  accumulate  the  execution  of  1,250,000 
randomly  generated  test  cases. 

The  recurring  cost  figures  are  calculated  assuming  that  no  errors  are 
found  in  the  unit  under  test.  If  any  errors  are  found,  this  figure 
would  increase  proportionally.  In  all  the  remaining  cases,  therefore, 
the  recurring  figures  are  all  within  a  one  week  time  frame. 

The  second  component  of  recurring  cost  is  the  scf tware/hardware 
maintenance  cost  associated  with  each  approach.  The  Lockstep  approach 
with  the  "golden"  computer  distinguishes  itself  as  being  the  only 
approach  which  has  hardware  maintenance  cost.  Software  aaintenance 
costs  are  proportional  to  the  size  cf  the  verification  program. 


7.4.4  Time  fiequired  to  Fe  rf orm  Valida  ticn 


The  time  required  to  perform  vecificaticn  has  been  defined  to  be  the 
time  between  when  the  verification  program  is  initially  loaded  into 
the  unit  under  test,  and  when  it  successfully  completes  execution  (no 
errors  found).  As  can  be  seen  frcm  Table  7-27,  only  cne  approach’s 
execution  time  is  large  enough  to  immediately  eliminate  it  from 
fcrther  consideration,  that  is  the  Random  Instruction  approach  in  a 
Manual  Test  Configuration.  All  other  approaches  fall  well  within  a 
cne  week  test  period  time  frame. 
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■? .  9 . 5  Impact  to  SEAF A C  Resources 


1^€  basic  resources  that  are  commoii  to  all  approaches  and  test 
configurations  are  the  fcllowing: 

•  Development  Computer 

•  Support  Software 

ilIL-STD-1750  Cross  Assembler 

Linking  Loader 

Simulator 

•  Development  Personnel 

•  Certification  Facility 

•  Certification  Plan  Document 

•  Certification  Personnel 

•  Source  Tape  Dump  Program 

Tie  resources  which  apply  to  the  Master/Slave  Test  Configuration  are: 

•  Bootstrap  Load  Program 

•  Control  Program  on  Master 


•  I/O  Test  Program 

The  Lockstep  approach  requires  the  "golden"  computer  to  be  purchased, 
verified  and  maintained  which  adds  appreciable  ccst. 
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7.U.6  Quality  Ex£ectaticns  for  Icrificaticn  Fe thoU s 


lihile  the  foregcin>j  quality  analysis  has  rot  resulted  in  surricient 
confidence  to  discriminate  between  the  verification  methods  on  the 
basis  of  quality,  some  judgements  can  he  made  about  the  expected 
differences. 

First  of  all,  it  is  likely  that  there  are  not  significant  differences 
ir  quality  between  the  Functicnal  approach  and  the  Lockstep  approacn. 
Fach  of  these  relies  on  manually  generated  test  cases.  Assuming  equal 
ability  and  insight  of  the  people  writing  the  test  cases,  tr.ere  would 
be  no  differences  in  the  quality  of  the  test  cases. 

The  two  approaches  do  differ  in  the  method  of  generating  the  expected 
results.  In  the  Functicnal  approach  they  are  generated  manually  and 
ir  the  Lcckstep  approach  they  are  generated  by  the  "golden"  machine. 
Since  the  expected  results  are  the  same  in  bcth  cases,  there  should  he 
nc  impact  on  quality. 

The  two  approaches  may  also  differ  in  terms  of  the  checking  of 
architectural  entities  (e.g.,  registers,  indicators)  wnich  should 
remain  unchanged  by  a  test.  It  is  relatively  easy  for  the  programmer 
using  the  Lockstep  approach  to  check  all  the  status  information  and 
all  the  general  registers  and  all  the  address  registers  for  any 
urdesired  changes.  To  do  this,  he  needs  to  keep  track  of  the  contents 
of  these  registers.  To  accomplish  the  same  checks  using  the 
Functional  approach  requires  that  the  programmer  maintain  tne  expected 
information.  To  the  degree  that  the  programmer  using  the  functional 
method  does  not  check  all  of  the  architectural  entities  for  lack  of 
change,  the  Lockstep  method  may  offer  a  slightly  higher  quality. 
Eowever,  the  differences  would  not  be  expected  to  be  significant. 

However,  a  comparison  between  the  Fandom  approach  and  either  the 
lockstep  approach  or  the  Functicnal  approach  does  reveal  an  expected 
difference  in  quality.  First,  consider  a  simple  hypothesis:  the 
quality  of  an  approach  is  a  function  of  both  the  number  of  test  cases 
and  the  "value"  of  those  test  cases.  This  appears  to  agree  with 
intuition.  Each  additional  test  case  provides  some  improvement  in  the 
Quality  of  the  verification  method  (assuming  it  is  not  merely  a  copy 
of  a  previous  test  case). 

Some  test  cases  may  be  more  valuable  than  others  for  improving  the 
quality  of  a  verification  approach.  It  would  be  expected  that  a 
collection  of  one  hundred  carefully  thought-out  test  cases  would  be  a 
better  check  of  a  machine  than  one  hundred  randomly  chosen  test  cases 
(which  may  not,  for  instance,  even  check  all  addressing  modes  or  even 
the  ADE  instruction) .  For  this  reason  the  Functional  type  of 
verification  approach  would  be  expected  to  be  of  higher  quality  than 
the  Fandom  approach  for  the  same  number  of  test  cases. 

The  comparison  does  not  end  here,  however,  since  the  number  of  test 
cases  must  be  taken  into  consideration.  Since  the  test  cases  for  the 
Functional  type  of  verification  approach  must  be  generated  manually, 
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the  cost  cf  generating  each  additional  test  case  is  significant.  A 
doubling  of  the  number  cf  test  cases  %ould  result  in  a  near  doubling 
cf  the  cost.  For  the  Random  approach,  hovever,  in  which  the  test 
cases  are  automatically  generated,  the  cost  cf  generating  each 
additional  test  case  is  very  small.  A  doubling  of  the  number  of  test 
cases  would  result  in  only  a  small  increase  in  total  ccst. 

The  result  is  that  it  is  not  clear  which  approach  would  provide  the 
greater  quality  test.  The  answer  will  clearly  be  a  function  of  the 
duration  of  the  test.  Given  the  opportunity  to  run  Icng  enough,  the 
Random  test  will  surpass  the  Functional  test  in  terms  of  quality 
(since  the  Functional  test  is  fixed  in  terms  of  the  number  of  test 
cases) .  The  break-even  point  in  terms  of  time  is  not  known,  however. 

Ar.  anecdote  regarding  the  ccmpariscn  cf  the  Functional  and  Random 
approaches  may  be  of  interest-  Two  different  machines  were  developed 
by  the  lEM  System  Products  Division  at  Poughkeepsie,  N.Y.  The  first 
machine  was  developed  using  a  Functional  type  of  test.  A  Random  type 
test  was  then  applied,  and  some  additional  architectural  discrepancies 
were  discovered.  The  second  machine  was  developed  using  the  Random 
type  of  test.  A  Functional  test  was  then  applied,  and  no  additional 
architectural  discre pa ncies  were  found.  This  suggests  that  the  Random 
test  is  of  higher  quality. 
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7.^  INTiHPRETATION  OF  CCS!  NCDEL  EESOITS 


Since  the  quality  analysis  was  unable  tc  provide  a  credible  cost 
penalty  xigure  (due  to  less-than-per feet  Quality)  fer  each  or  tr.e 
veri f icaticn  approaches,  these  costs  cannot  be  considered  rn  the  cost 
mcdel-  This  does  not,  however,  reduce  either  the  reality  or  the 
significance  of  these  costs.  Neither  is  the  quality  analysis 
ccmpleted  invalidated. 

Ignoring,  for  the  aoraent,  the  cost  penalty  of  having  a  less  than 
perfect  verirication  approach,  cne  is  left  with  the  conclusion  that 
the  lowest  cost  verification  method  (as  shewn  in  Table  7-26)  is  tr.e 
test  one.  This  is  simply  because  there  is  no  firm  data  to 
differentiate  between  the  methods  in  terms  of  Quality.  Ihe 
modification  of  tne  AN/AYK-15A  Acceptance  Test  Procedure  (ATP)  is  the 
lowest  cost  and,  therefore,  the  best  method  (in  the  absence  of  quality 
considerations).  The  previous  comments  about  the  availability  of  an 
FTP  eliminate  the  FTP  approach,  which  has  the  same  cost  of 
modification  of  tne  AN/AYK-15A  ATP  approach. 

The  quality  analysis  does  provide  valuable  insiqht  into  the  cost 
penalties  associated  with  this  method,  however. 

The  AN/AYK-15A  ATP  is  of  the  Functional  generic  type,  a.s  discussed 
Etsvicusly.  As  described  in  Secticn  7. 3.2.5  sufficient  data  do  exist 
tc  provide  a  rough  cost  estimate  (due  tc  less  than  perfect  quality)  of 
the  Functicnal  type  of  verification  method. 

The  results  of  that  section  indicate  that  the  expected  cost  penalty 
that  would  be  paid  for  having  a  less- thar-per feet  verification  method 
(in  this  case,  the  Functional  type)  is  between  $2.8  and  34.6  million- 

This  is  a  significant  cost.  Obviously,  if  it  were  possible  to 
significantly  improve  the  quality  of  the  overall  test  at  a  cost 
significantly  less  than  the  $2.8  to  $4.6  million  range,  this  would  be 
very  desirable.  The  data  gathered  strongly  suggest  that  the  Random 
approach  to  architectural  verification  is  the  best  method  of 
augmenting  the  quality  of  the  AN/AYK-15A  ATP  approach. 

Three  reasons  apply.  First,  the  Random  approach  provides  an  extremely 
large  number  of  test  cases,  many  times  the  number  of  test  cases  of  the 
AN/AYK-15A  ATP  approach.  Each  new  test  case  provides  a  positive 
increment  to  the  overall  quality. 

Second,  the  test  cases  generated  under  the  Random  approach  are 
expected  to  be  independent  of  (i.e.,  net  related  to)  the  test  cases 
used  in  the  AN/AYK-15A  ATP  approach.  Tliis  is  simply  because  in  the 
Fandcm  approach  they  are  generated  randomly  and  in  the  AN/AYK-ISA  AT? 
approach  they  ace  generated  manually.  The  other  verification  methods 
examined  in  this  study  also  have  manually  generated  cest  cases.  They 
would,  therefore,  be  expected  to  he  less  independent.  This 
independence  is  required  in  order  to  obtain  improvements  in  quality. 
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Third,  the  Randcm  approach  provides,  by  far,  the  largest 
test  cases  per  dollar  expended. 


rutther  of 


The  two  methods  added  together  should,  therefore,  imprcve  the  total 
grality.  The  anecdote  in  the  previous  section  implies  that  the  Random 
approach  offers  a  higher  quality.  Furthermore,  testing  at  SEAFAC 
should  permit  a  Random  test  of  long  duration,  since  overnight  and 
weekend  testing  is  possible.  About  28.8  million  test  cases  should  be 
possible  in  a  week  long  verification  test. 


The  cost  of  implementing  the  Random  certification  method,  as  shown  in 
Table  7-28,  is  well  below  the  $2.8  to  $4.6  millicn  range  discussed 
above.  Ix  the  overall  quality  of  the  test  were  improved  by  22  percent 
by  the  addition  of  the  Random  approach,  then  the  reduction  of  the  cost 
penalty  by  $622,000  =  ($1,239,000  -  617,000)  (22  percent  of  $2.8 
million)  would  offset  the  cost  of  implementing  the  Randcm  verification 
method.  It  seems  reasonable  to  expect  that  the  addition  of  26.6 
million  test  cases  to  the  approximately  five  thousand  test  cases  in 
the  AN/AYK-15A  ATP  method  would  result  in  a  quality  improvement  of  22 
percent.  ($1,239,000  is  the  total  cost  for  the  IRK  recommendation 
which  will  be  discussed  later.)  While  the  data  gathered  in  this  study 
cannot  definitively  prove  it  to  be  the  case,  it  appears  that 
investment  in  the  combination  Functional/Fandom  approach  will  be 
easily  recovered. 


As  was  mentioned  previously,  caution  must  be  used  when  drawing 
ccnclusions  based  on  the  $2.8  million  to  $4.6  million  cost  penalty. 
However,  there  is  good  reason  to  believe  that  any  errors  will  be  on 
tfe  side  of  conservatism.  This  is  because  the  actual  cost  penalty  to 
the  Air  force  is  likely  to  be  substantially  higher  than  the  $2.8 
million  stated.  If  this  is  the  case,  then  the  justification  for 
adding  the  Random  approach  is  further  reinfcrced. 


Consider  again  tne  quality  analysis.  The  cost  penalties  measured  were 
for  the  correction  of  architectural  discrepancies  discovered  during 
C perat ional  Flight  Program  validation.  The  costs  here  are  limited  by 
the  fact  that,  since  this  is  still  in  the  development  phase,  changes 
crly  need  to  be  made  to  the  development  hardware  cr  software,  not  to 
many  machines  in  the  field.  Rut,  just  as  some  architectural 
discrerancies  escape  detection  during  architecture  verification,  some 
architt  turai  discrepancies  will  escape  detection  during  software 
validation.  Experience  shows  that,  even  after  years  of  operational 
use,  additicnal  architectural  discrepancies  can  be  uiscovered.  Inis 
cculd  bp  for  many  reasons;  previously  untried  data  cerbinations  are 
certainly  cne  possible  cause. 


Ccrrection  of  architectural  discrepancies  which  are  discovered  after 
operational  use  of  the  hardware  and  software  has  begun  can  cost 
several  order  of  magnitude  more  than  ccrrection  cf  those  same 
architectural  discrepancies  caught  during  the  development  phase  of  a 
nregtam.  This  is  simply  due  to  the  tremendous  Icgistics  involved  in 
chancing  many  computers  which  are  in  the  field.  Experience  shows  that 
these  costs  could  easily  overshadow  the  $2.8  to  $4.6  million  estimatt 
made  above.  Thio  further  emphasises  the  value  of  improving  the 
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quality  of  the  verification  test  to  reduce  the  large  cost  penalties 
vtich  result  from  havirg  •  “s-then-perfect  verif icatior . 

Naturally,  the  question  .  ‘‘  be  asked:  If  the  Eandom  approach  is  of 
higher  quality,  why  not  elimi'^ate  the  Functional  approach?  The  answer 
is  twc-fold.  First,  the  combination  of  the  approaches  provides  an 
even  higher  quality.  Second,  and  perhaps  the  more  irportant  to  the 
Air  Force,  the  Functional  approach  provides  a  certain  minimuffi  test  of 
the  machine.  The  Random  approach,  while  providing  perhaps  a  higher 
decree  of  quality  of  test,  cannot  provide  any  minimum  degree  of 
testing.  For  example,  it  is  theoretically  possible  that,  after  a 
nillicn  test  cases,  that  the  ADD  instruction  would  remain  untested. 
While  this  is  highly  unlikely,  the  ability  tc  provide  a  specific  level 
of  testing  must  be  of  considerable  importance  to  the  Air  Force. 

Another  question  may  be  asked  as  tc  why  this  combination 
Functional/Random  approach  has  not  been  applied  tc  avionics  machines 
in  the  past.  The  answer  is  based  on  simple  economics:  for  single 
applications  of  a  computer  architecture,  the  test  investment  must  be 
recovered  in  that  single  application;  multiple  use  architectures  like 
SIL-STD- 1750  can  recover  the  test  investment  over  those  multiple 
applications.  Effectively,  there  are  eccncmies  of  scale  in  applying 
the  test  over  and  over  again. 

The  quality  analysis  in  this  report  provides  some  benefits  to  the  Air 
Force  in  addition  to  helping  select  the  test  verification  approach. 
It  should  assist  the  Air  Force  in  developing  realistic  expectations 
regarding  the  certification  process  and  the  use  of  HIL-STD-1750 
machines.  The  collected  data  show  that  the  verification  method  will 
net  be  perfect.  Machines  will  be  certified  that  have  hidden 
architectural  discrepancies.  Previously  validated  software  will  not 
always  execute  correctly  cn  newly  certified  machines. 

Fcr  the  users  of  illL-STD- 1750  machines,  it  provides  a  conceptual 
framework  for  the  error  discovery  process  during  Operational  Flight 
Program  validation.  Understanding  of  these  results  shculd  permit  more 
accurate  budgeting  and  scheduling  for  programs  that  use  HIL-STD-1750 
computers. 

In  summary,  the  cost  model  has  provided  valuable  data  for 
discriminating  between  the  different  verification  approaches.  The 
quality  analysis,  while  unable  to  provide  high  confidence  in  the 
specific  dollar  penalties  to  be  assessed  each  verification  approach, 
has  provided  an  estimate  of  the  penalty  tc  be  paid  for  the  use  of  a 
single  verification  test  (in  this  case  Functional)  approach.  Ibis 
information,  combined  with  the  cost  data,  provide  the  basis  for  the 
recommendation  of  a  combined  Functional/Fandom  approach. 

This  recommendation  will  be  discussed  in  detail  in  the  following 
section. 
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8 . 0  BECCMHFMDAIIOM 


This  section  contains  IBS’s  recommendation  to  the  Air  force  for 
implementing  the  dlL-STD- 1750  certification  capability  in  SEAFAC.  It 
consists  of  a  description  of  the  approach,  and  the  rationale  leading 
to  its  selection.  A  summary  of  the  resources  required  is  also 
covered.  As  a  conclusion,  an  outline  for  the  llIl-STD-1750 
Certification  Test  Plan  which  describes  the  chain  of  events  leading  up 
to  vendor  certification  is  presented. 


8.1  TBO  LEVEL  APPBOACH 


8.1.1  Description 


IBM  recommends  a  two  phase  approach  to  certification  as  follows: 

Phase  1  A  deterministic  verification  approach  based  on 

modifications  to  the  AB/AYK-15A  ATE  is  run  on  the 
MIL-STD-1750  computer  being  tested.  (These 
modifications  delete  the  non-MIL-SlE- 1750  features 
from  this  ATP.)  This  provides  a  predefined 
minimum  test  of  the  machine  using  5,000  test 
cases.  This  first  phase  tests  the  integrity  of 
each  MIL-STD- 1750  instruction  for  selected  data 
patterns  and  addressing  modes  as  described  in  the 
Military  Standard  Airborne  Computer  Instruction 

Set  Architecture  document  and  tests  the  boundary 
problem  areas  in  the  HIL-STD-1750  architecture. 

Phase  2  After  completion  of  Phase  1,  a  Bandcm  verification 

approach  is  used  to  generate  large  numbers  of  test 
cases.  The  second  phase  of  testirg  verifies  that 
large  number  of  instruction  sequences  execute 
according  to  their  intended  function.  A  test  case 
contains  an  instruction  which  is  first  run  on  a 
MIL-STD-1750  simulator  and  ther  run  on  the 
MIL-STO-1750  computer  being  tested.  The  simulator 
results  are  compared  with  the  results  from  the 
MIL-STD-1750  computer  being  tested.  28,800,000 

test  cases  should  be  possible  during  a  week  long 
verification  test. 

Both  phases  of  testing  would  be  conducted  under  a  Haster/Slave  Test 
Configuration.  The  MIL-STD-1553  Serial  Channel  I/C  interface  is 

recommended  as  being  the  standard  to  which  each  vendor  should  target 

his  certification  configuration;  but  a  multiple  interface  capability 
is  recommended  ~  MIL-STD-1553  or  PS-232.  Ramifications  of  using  a 
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ES-232  Serial  Channel  1/0  interface  are  identified  in  Section  7. 

Tlis  provides  a  superior  means  to  accomplish  the  certification  process 
hecause: 

1.  Ihe  2>phase  process  (with  Bandom  as  the  second  phase) 
provides  a  higher  quality  than  any  single  verification 
approach. 

2.  Ihe  2-phase  process  provides  a  minimum  testing  level. 

The  recommended  approach  for  arriving  at  a  verification  program  that 
fulfills  the  requirements  of  the  first  phase  cf  testing,  is 
accomplished  by  the  modification  of  the  AH/iYK-15R  ATP.  The 
recommended  approach  for  arriving  at  a  verification  program  that 
satisfies  the  requirements  of  the  second  phase  of  testing  is  the 
development  of  a  Bandcm  Instruction  Generator  Pregram,  and  the 
modification  of  the  existing  KIL-STD-1750  simulator  to  interface  with 
a  control  program  that  compares  the  results. 

The  verification  testing  can  be  achieved  in  less  that  two  weeks  of 
time. 


e . 1 . 2  Bationale 


The  rationale  for  selection  of  a  two  level  approach  tc  architectural 
verification  was  discussed  in  detail  in  Section  7.  This  rationale  is 
summarized  here. 

The  cost  model  shows  that  the  lowest  cost  approach  is  the  modification 
cf  the  existing  AN/AYK-15A  Acceptance  Test  Program.  This  method, 
then,  constitutes  the  first  level  of  verification  for  MIL-STD-1750 
machines. 

The  quality  analysis  shows  that  use  of  this  single  method  results  in  a 
cost  penalty  to  the  Air  Force  of  tetween  $2.8  and  $4.6  million.  This 
oenalty  is  the  additional  software  validation  costs  that  is  incurred 
over  the  lifetime  of  MIL-STD-1750  because  of  the  less-than-per feet 
quality  of  the  AN/AYK-15A  ATP.  The  magnitude  cf  this  cest  suggests 
that  use  of  an  additional  method  of  verification  testing  to  improve 
the  overall  quality  could  substantially  reduce  the  overall  cost  and  at 
the  same  time  provide  SEAFAC  with  a  tetter  test. 

The  data  gathered  strongly  suggest  that  the  Bandcm  approach  to 
architectural  verification  is  the  best  method  of  augmenting  the 
quality  of  the  AN/ATK-15A  ATP.  Three  reasons  apply.  First,  the 
Bandcm  approach  provides  an  extremely  large  number  of  test  cases,  many 
times  the  number  of  test  cases  of  the  AB/AYK-15A  ATP.  Each  new  test 
case  provides  a  positive  increment  to  the  overall  quality. 
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Second,  the  test  cases  generated  under  the  Fandcii  approach  are 
expected  to  be  independent  of  (i.e.,  net  related  to)  the  test  cases 
used  in  the  AN/AXK-15A  ATP.  This  is  siirply  because  in  the  Sandom 
approach  they  are  generated  randomly  and  in  the  AN/AYK-ISA  ATP 
approach  they  are  generated  manually.  The  other  verification  methods 
examined  in  this  study  also  have  manually  generated  test  cases.  They 
vculd,  therefore,  be  expected  to  he  less  independent.  This 
independence  is  required  in  order  to  obtain  improvements  in  quality. 

Third,  the  Bandcm  approach  provides,  by  far,  the  largest  number  of 
test  cases  per  dollar.  It  is  reasonable  tc  expect  that  the  additional 
cost  cf  the  Random  approach  ($622,000)  is  offset  by  a  reduction  in  the 
cest  nenalty  due  to  the  improvement  in  quality.  This  requires  an 
improvement  in  quality  of  22  percent.  It  seems  reascrable  tc  expect 
that  increasing  the  number  of  test  cases  from  5,000  to  28,800,000 
would  provide,  at  least,  a  22  percent  improvement  in  quality,  although 
there  is  no  firm  data  to  support  this. 

Cemments  about  the  two  level  approach  with  regards  tc  the  evaluation 
criteria  follow. 


OOAIITY 

The  two  level  approach  affords  a  higher  degree  cf  quality  than  any 
single  approach.  The  two  levels  are  complementary  in  that  tne 
AN/AYK-15A  ATP  approach  provides  a  predetermined  minimum  test  of  all 
instructions  (but  not  a  large  number  of  test  cases)  and  the  Random 
approach  provides  a  large  number  of  test  cases  (but  does  not  provide 
any  certainty  for  testing  all  instructions)  . 

The  test  cases  from  Random  are  expected  to  be  independent  from  the 
test  cases  generated  by  the  ATP.  That  is  simply  because  in  Random 
they  are  generated  randomly  and  in  the  AN/AYK-151  ATP  they  are 
generated  manually.  This  independence  is  required  for  an  improvement 
it  quality. 

The  quality  of  the  certification  approach  can  be  increased  by  allowing 
the  Fandom  instruction  verification  program  to  run  for  longer  periods 
cf  time.  Under  the  Haster/Slave  Test  Configuration,  the  Random 
instruction  verification  program  can  be  left  to  run  unattended  during 
off  hours  (on  second  and  third  shifts  or  over  the  weekend). 

The  Non-Recurring  Start-Op  cost  for  ivplementaticn  of  the  IBM 
recommendation  within  a  year  and  a  half  is  $933,000;  the  Recurring 
cost  for  30  computers  over  10  years  is  $306,000;  the  total  cost  is 
thus  $1,239,000  for  the  implementation  of  the  recommendation. 

The  AN/AYK-15A  ATP  approach  was  shewn  by  the  cost  acdel  to  be  the 
lowest  cost  approach.  Addition  of  the  Fandom  approach  will  increase 
the  cost  to  SEAFAC,  but  should  result  in  an  overall  cost  reduction  to 
the  Air  Force  because  of  the  increase  in  quality. 
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TEST  CONFIGDEATICB 

The  Kaster/Slave  Test  Configuration  is  reccirmended.  The  Master/Slave 
Test  Configuration  provides  tbe  only  feasible  environniert  in  which  the 
Bandom  Instruction  Verification  approach  exists.  However,  the 
AH/AIK-15A  ATP  approach  tuns  under  either  a  Manual  or  Baster/Slave 
Test  Configuration. 

VODOR  IMPACT 

The  AN/AYK-15A  ATP  approach  will  yield  a  verification  program  of  which 
each  vendor  could  utilize  portions  to  pretest  his  MIl-srD-1750  prior 
to  certification  provided  that  the  source  program  he  made  available  to 
the  vendor.  This  availability  will  decrease  the  likelihood  of  finding 
"first  order"  errors  in  the  unit  under  test.  Additionally,  the  Random 
verification  approach's  Generator  Program  and  Simulator  Program  could 
also  be  used  by  the  vendor  to  further  pretest  his  MIL-STD-1750 
computer  prior  to  the  certification  process  in  order  to  increase 
confidence  in  passing  the  certification  process. 

IMPACT  TO  SFAFAC 

SEAFAC's  VAI  11/780  can  be  used  as  the  Master  computer  for  the 
certification  process.  Support  software  would  be  available  on  it  for 
develcpmental  and  maintenance  activities.  SEAFAC  personnel  (an 
observer,  a  technician  and  a  coordinator)  is  needed  during  the 
certification  process.  One  programmer  is  needed  tc  sustain  the 
Acceptance  Test  Program  and  the  Random  programs;  also,  five  software 
programmers  are  required  to  develop  the  Fandom  program  (within  one 
year)  unless  this  is  subcontracted  outside  of  SEAFAC.  If  the  major 
software  components  resident  on  the  Master  Computer  (Simulator,  Random 
Instruction  Generator)  are  written  in  an  HCL,  the  vendor  is  able  to 
irstall  them  on  a  computer  system  other  than  the  VAX  117/80,  thus 
facilitating  pretesting  at  the  vendor's  facility. 


VEBIFICATICN  TEST  TIME 

The  certification  process  takes  less  than  two  weeks  for  a  500  KOP 
computer  using  a  MIL-STE-1553  channel  tc  a  VAX  11/780.  (It  takes  7 
full  days;  but  two  weeks  should  be  allocated  for  the  certification 
process.)  If  a  150  FOP  computer  (not  a  500  KOP  computer)  is 
submitted,  the  time  for  certification  expands  less  than  10  percent 
(frcit  7  tc  7.7  days)  ,  but  remains  less  than  two  weeks.  If  a  500  KOP 
ccmruter  is  submitted  with  an  ES-232  interface,  the  time  for  the 
certification  process  expands  17  percent  (frcm  7  full  days  tc  8.2  full 
days)  if  concurrent  overlapped  CFO  and  I/O  processing  is  not  permitted 
in  the  VAX  11/780;  but  remains  less  than  two  weeks.  If  concurrent 
overlapped  CPU  and  I/O  processing  is  permitted,  the  CPU  processing 
still  dominates  and  completed  hides  the  I/O  transfer.  In  this  case, 
the  ES-232  channel  would  have  nc  discernable  impact  to  the  tine 
required  for  the  certification  process. 
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IS“2  IZ2  IRTEBFACI 

With  the  MIL-STD- 1553  Serial  Channel  interface  planted  for  the  VAX 
11/780,  it  should  be  used  in  the  Master  computer  rcle  during  the 
ccnduction  of  the  certification  process  to  connect  to  the  computer 
teing  tested.  (Since  the  MIL-STD-1750  Serial  I/O  Channel  already 
exists  on  the  POP  11/55,  the  POP  11/55  could  bridge  the  time  until  the 
VAX  11/780  system  with  its  MIL-STD-1553  I/O  channel  is  available.) 
Ihe  HIL-STO-1553  Serial  Channel  requirement  is  considered  to  have  the 
least  impact  on  the  vendor's  hardware  configuration  since  it  is 
expected  that  from  90  to  95  percent  of  of  the  computers  being 
submitted  for  certification  will  contain  a  MIL-STi:-1553  channel. 
However,  an  RS-232  serial  channel  is  also  recommended  for  those 
computers  without  the  MIL-STD-1553  channel. 

Parenthetically,  any  parallel  channel  with  a  transfer  rate  of 
approximately  1  mega  words  per  second  is  rejected  because  it  speeds  up 
verification  about  2  percent  from  7  to  6.87  days  and  is  not  a 
significant  improvement.  That  is,  the  verification  process  is  not  I/O 
hound,  but  is  CPU  bound  by  the  VAX  11/780. 


’ll 
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€.2  IMPLEaZNTATION  CONSIDEEATIONS 


ll€  folloHing  section  outlines  the  najOE  items  surcounding  the 
i irpl€B€ntation  of  the  two  level  certifies ticn  process. 


6.2. 1  Phase  1  ;  AM/AYK-  15A  ATP  Bodification 


This  Acceptance  Test  Plan  (ATP)  for  the  AN/AYK-15A  consists  of  the 
fcllcwing  subtests: 

•  User's  Console 

•  Instruction  Set 

•  fiegister 

•  Bain  Storage 

•  Eus  Controller 

•  Input/Output 

•  Power  ON^OFF  Sequencing 

An  ATP  resident  on  the  AN/AYK-15A  is  executed  according  to  procedures 
described  in  Air  Force  document,  PA  401  207,  in  which  a  user  interacts 
with  the  console  keyboard.  Each  suhtest  may  require  a  separate 
processor  load.  A  standalone  test  is  one  in  which  only  the  processor 
and  an  attached  console  (keyboard  and  CPT  display)  are  required  in 
erder  to  run  the  ATP.  The  Eus  Ccntrcller  Test  is  net  a  standalone 
test  in  that  it  requires  a  configuration  which  includes  the  processor 
under  test  and  at  least  one  other  Baster/Pemote  device  interfaced  via 
a  multiplex  bus.  Also,  various  tests  require  use  of  external  test 
equipment  (e.g. ,  logic  analyzers,  oscillcscopes,  etc.)  and/or  special 
purpose  Input/output  exercisers. 

In  order  to  develop  a  verification  program  to  fulfill  the  requirements 
cl  the  first  phase  of  testing,  modifications  to  beth  the  control 
picgram  and  subtests  of  the  AN/AYK-15A  Acceptance  Test  Plan  are 
necessary.  These  modifications  are  summarized  in  Table  8-1. 


8-6 


6176175A 


FINAL  BEPCPI 


Fetruary  29,  1980 


Table  8-1.  MIL-STD- 1750  Applicability  of  ATP  Suttests 


+ - - - + 


1  AN/AYK-15A 

1  Subtest 

i  Purpose 

HIL-STE-1750 

Applicability 

1 Keytoard/CET 

1 

I 

1 

I 

(Test  all  keyboard/CBT 

1  keystrokes. 

(Print  all  keystrokes  on 
|CBT  (keyboard  and  CBT  in 
(Local  mode). 

Deleted  for  tlIL-STD-1750 
Ve rification 

t 

lOsec  Coniffland 

1 

1 

1 

(Verify  correct  execution 
(of  all  user  ccmnands. 
(Exercise  all  defined 
(user  commands. 

Deleted  fcr  MIL-STD-175U 
Verification 

1 

(Floppy  Disc/ 

1  Printer 

1 

1 

1 

1 

1 

(Verify  proper  operation; 
(response  to  user 
(commands. 

("Wrap”  processor  main 
(storage  frcm  floppy  disk 
(to  printer  via  user 
( commands. 

Deleted  for  MIL-STD- 1750 
Verification 

1 

(Basic  Instruction 
(  (done  for  each 
(  instruction) 

1 

1 

1 

(Test  Basic  Instructicn 
(Operation. 

(Exercise  Instruction  with 
(known  input  parameters 
(and  ccmpare  with  expected 
( results. 

Modify  for  MIL-SID-1750 
Verification  as 
necessary 

1  “ 
(  Aritboietic 
(Instruction  Test 

1 

1 

1 

( 

1 

(Test  all  arithmetic 
(instructions  and  status. 
(Exercise  all  arithmetic 
(instructions  using 
(selected  values  in  the  4 
(quadrants;  check  condi- 
(tion  status  setting. 

Modify  for  MIL-STD-1750 
Verification  as 
necessary 

I 

(Condition  Status 
i  (done  for  each 
(  application) 

1 

1 

1 

1 

(Verify  setting  of  the 
(status  word  for  each 
(instruction. 

(Cause  result  of  exercised 
(instruction  to  set  all 
(required  ccmbinations  of 
(status  bits. 

Modify  fcr  MIL-STD-1750 
Verification  as 
necessary 
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Table  8-1.  lSIL-STD-1750  Applicability  of  ATP  Subtests  (cout) 


4. - - - -  4 


AN/AYK-15A 

Subtest 

1  f 

i  Purpose  1 

BIL-ETE-1750 

Applicability 

Indexed 

Addressing 
(done  for  each 
applicable 
instruction) 

(Test  those  instructions  | 
(using  index  capability.  j 
jose  15  index  registers;  | 

1  positive  and  negative  j 
(indexing.  ( 

Modify  for  MlL-STD-1750 
Verification  as 
necessar y 

Cverf low/ 
Underflow 
(done  for  each 
instruction) 

(Test  those  instructions  ( 
(which  cause  underflow/  ( 
(overflow;  verify  under-  j 
(flow/overflow  interrupts.) 
(Induce  underf low/overf low ) 
(resultant.  | 

Modify  for  MIL-STD-1750 
Verification  as 
necessary 

Eencbmark 

(Measure  processor  ) 
(throughput.  ) 
(Use  benchmark  arithmetic  ) 
(and  logic  instructions  | 
(mixes  and  measure  average) 
(processor  throughput.  | 

Deleted  for  MIL-STD-1750 
Verification 

Rang  Test 

(Check  for  illegal  ) 
(sequences  cf  I 
(instructions.  ) 
(Use  random  sequence  of  | 
(instructions  and  test  ) 
(for  invalid  indicators.  ) 

Deleted  for  MIL-STD-1750 
Verification 

Illegal 

Instruction 

(Check  fault  register  | 
(setting,  machine  error  ) 
(interrupt,  clear  fault  ( 
(register  output  command.  ( 
(Execute  illegal  instruc-  | 
(tion  and  test  for  I 
(expected  response.  1 

Modify  for  MIL-STD-1750 
Verification  as 
necessary 

General  Register 

(Testability  to  address  ( 
(and  set/reset  all  16  ( 
(general  registers.  ( 
(Wrap  known  patterns  from  ( 
(processor  main  storage.  ( 

Modify  for  MIL-STD-1750 
Verification  as 
necessary 
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Table  8-1.  illL-STD- 17 50  Applicability  of  ATP  Subtests  (cont) 


1  AN/AYK-15A 

1  Subtest 

1 

1  Purpose 

HIL-STD-1750 

Applicability 

lECM  Control 

1 Begisters 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1  Testability  to  address 
land  set/reset  all  16  ECM 
Icontrcl  registers;  verify 
land  reset  the  test  hit, 
icompare  register  and 

Istore  register  input  ard 
loutput  commands. 

1  Wrap  known  data  patterns 
(from  the  processor 
(general  registers  via 
linput  and  output 

1 instructions. 

Deleted  for  flIL-STD-1750 
Verification 

1 

|ECM  General 

1 Begisters 

1 

1 

1 

1 

1 

1 

(Testability  to  address 
land  set/reset  all  16  ECU 
Igeneral  registers. 

(Wrap  known  data  patterns 

1  from  the  processor 
(general  registers  via 
linput  and  output 

1 instructions. 

Deleted  for  hlL-STD- 1750 

Ve ri f ication 

1  " "  ^ 

1  Status  Begxster 

1 

1 

1 

1 

1 

1  Testability  to  set/reset 
(the  status  register. 

(Wrap  known  data  patterns 
(from  general  registers 
(via  input/output 

1  instructions. 

Modify  fcr  MIL-STD-1750 
Verification  as 
necessary 

\ 

1  Instruction 
ICounter 

I 

1 

1 

1 

(Testability  to  sequence 

1 instructicn  counter 
(through  64K  main  storage. 
(Increment  counter  and 
(verify  execution  of  every 
(increment  instruction. 

Modify  fcr  MIL-STD-1750 
Verification  as 
necessary 

1  ■  ■ 

1  Interrupt  UasJc 

1 

1 

1 

1 

1 

(Testability  to  set/reset 
(interrupt  mask  register. 
(Wrap  known  data  patterns 
(from  general  registers 
(via  input/output 
( instructions. 

Modify  fcr  MIL-STD-1750 
Verification  as 
necessary 
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Table  8-1-  MIL-STD- 1750  Applicatilit v 


ATP  Subtesta  (cent) 


AN/AYK-15A 

Subtest 


Krite  Protect 
EAM 


Main  Storage 
Integrity 


Processor  CPU 
Vlrite  Protect 


Pead  Only 
Memory  (P.CM) 


Purpose 


Illegal  Main 
Storaae  Address 


Main  Storage 
Access 


Testability  to  set/reset 
write  protect  RAM. 

Wrap  known  data  patterns 
from  general  registers 
via  input/output 
instructions. 


T'^stability  to  address, 
write  and  read  64K 
main  storage. 

Wrap  known  worst  case 
data  patterns  frem 
general  register  (1*e, 
O's,  and  addresses). 


Testability  to  protect 
main  storage  in 
increments  of  IK  blocks. 
Test  main  storage 
protect  enable  output 
command;  test  fault 
register  setting. 

Store  data  into  protected 
areas  and  compare  for 
expected  results. 


Test  ECM  integrity;  test 
enable  and  disable  output 
command  discretes. 

Perform  cyclic  checksum 
within  POM  memory  using 
ROM  enable/disatle  output 
instructions. 


Verify  setting  of  fault 
register  indicator. 
Remove  main  storage 
module  and  attempt  to 
read/write  vacant 
locations. 


Test  for  main  storage 
access  and  cycle  time. 
Instrument  main  storage 
controller  and  measure 
access  and  cycle  times. 


KIL-STt-1750 

Applicability 


Modify  fer  M1..-STD- 1750 
Verification  as 
necessary 


Modify  for  MIL-STD-1750 
Verification  as 
necessar  y 


Modify  for  MIL-ETD-1750 
Verification  as 
necessary 


Modify  for  MIL-STD-1750 
Verification  as 
necessary 

Optional  in  MIL-STD-1750 


Modify  fer  MIL-STD-1750 
Verification  as 
necessary 


Deleted  for  MIL-STD-1750 
Verification 
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Table  8-1. 

BIL-STD-1750  Applicability  of 

ATP  Subtests 

(cent) 

AN/AYK-15A 

Subtest 

1  1 

1  Purpose  1 

MIL-STE-1750 

Applicability 

Bain  Storage 
Access  Priority 

1  Verify  priority  of  Eus  | 
IContrcl  nodule  (ECU),  | 
{Direct  Memory  Access  | 

1  (DMA)  and  CPO  access  to  | 
{main  storage.  1 
{Initiate  simultaneous  { 
{main  storage  requests  and{ 
{instrument  main  storage  { 
{controller.  { 

Deleted  for 
Verification 

MIL-STD-1750 

ECn  Internal 

{Test  NOOP,  LINK  and  HALT  { 

Deleted  for 

MIL-STD-17S0 

Commands 

{BCM  instructions;  BCI  ( 
{Level  1  interrupt.  { 
{Exercise  instructions  in  ( 
{predefined  order  (stand-  { 
{alone  only).  { 

Verification 

ECM  Self-Test 

{Test  capability  of  ECM  | 
{and  MTO  to  wrap  self-test{ 
{pattern,  test  bus  ( 
(activity  discretes.  { 
{Exercise  BCM  self-test  j 
{function  (standalone  I 
(only)  .  { 

Deleted  for 
Verification 

BIL-STD-1750 

Undefined  Bode 

(Test  Master/Bemote  | 

Deleted  for 

MIL-STD-1750 

Commands 

(response  to  undefined  ( 
(mode  commands;  ECI  ( 
(Level  2  interrupt.  { 
(Exercise  predetermined  { 
(mode  command  message  ( 
(sequence  (requires  Master) 

{ and  Bemote) .  ( 

Verification 

BTO  Shutdcwn 

(Test  Master/Bemote  { 

Deleted  for 

MIL-STD-1750 

Bode  Commands 

(response  to  MTO  shutdcvn  j 
(mode  commands.  { 
(Exercise  predetermined  ( 
(mode  command  message  ( 
(sequence  (requires  Master) 
(and  Remote).  { 

Verification 

in 
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Table  8-1.  MIL-STD-1750  Applicability  cf  ATP  Subtests  (coct) 


1  AN/AYK-15A 

1  Suttest 

1 

1  Purpose 

MIL-saE-1750 

Applicability 

INode  Comfflands 
Iwith  Interrupts 

1 

1 

1 

1 

1 

1 

jTest  Master/Eemcte 
(response  to  those  node 

1  commands  vhich  generate 
(remote  interrupt. 

(Exercise  predetermined 
(mode  command  message 
(sequences  (requires 
(master  and  remote). 

Deleted  for  mi-STD- 1750 
Verification 

1  Synchronous  Data 

(Test  Haster/Eemote  data 

Deleted  for  HIL-STD-1750 

1  Transfers 

(addressing/  tag  word 

Verification 

1 

(storage,  data  storage. 

1 

(Exercise  predetermined 

1 

(message  sequences. 

1 

(rotating  word  count  and 

1 

( subaddress. 

1 - 

1  Asynchronous  DatalTest  Master/Pemote  ASYNC 

Deleted  for  HIL-STD-1750 

1  Transfers 

(data  addressing,  ASYNC 

Verification 

1 

(message  protocol. 

1 

(Exercise  predetermined 

1 

(message  sequences. 

1 

(rotating  word  count  and 

1 

1  subaddress. 

I  Timer 

(Test  capability  tc  load. 

Hodify  for  MIL-SID- 1 750 

1 

(start  and  stop  timers  A 

Verification  as 

1 

(and  E.  Test  capability 

necessary 

1 

(to  input  timers  A  and  E. 

1 

(Test  timer  A  and  E 

1 

( interrupts. 

1 

(Output  various  timer 

1 

(values  and  test  using 

1 

(known  instruction 

j 

(execution  times. 

1  External 

(Test  capability  to  set. 

Modify  fcr  MIL-siD-1750 

1  Eiscretes 

(reset  and  read  8  output/ 

Verification  as 

1 

(input  discretes;  Test 

necessary 

1 

(Read  Discrete  Output  and 

1 

(input/cutput  buffer  input 

1 

1  commands. 

1 

1  Wrap  discrete  output  to 

1 

(input,  output  fixed  data 

1 

(patterns  and  compare. 
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Table  8-1.  NIL-STD- 1750  Applicability  cf  ATP  Subtests  (cost) 


♦ 


1  AN/ATK-15A 

1  Subtest 

1  1 

1  Purpose  1 

MIL-STD-1750 

Applicability 

1  PIO 

1 

1 

1 

1 

1 

1 

1 

ITest  capability  to  | 

jexecute  PIO.  I 

1  1 
lOutput  to  PIO  and  ccmparel 

1  to  output  buffer  read;  | 

|use  external  test  hard-  | 
|ware  to  provide  external  | 
Iwrap.  1 

Modify  fcr  MIL-SIB-1750 
Verification  as 
necessary 

Optional  in  hlL-STE- 1750 

1  ■  ■ 

1  Interrupts 

1 

1 

1 

1 

1 

1 

1 

1 

1 

ITest  capability  to  | 
(respond  to  6  external  1 
(interrupts,  test  the  ( 
(clear,  disable,  enable,  | 
(and  clear  pending  ( 
(interrupts  output  ( 
(commands.  ( 
(Use  external  test  hard-  ( 

( ware  to  initiate  | 
(interrupts-  1 

Modify  fcr  MIl-STD-1750 
Verification  as 
necessary 

Optional  in  MIL-STD- 1750 

1  .  .  ..  I.  _  .  U  -1  -  -  - 

1  IMA 

1 

1 

1 

1 

1 

1 

(Test  DMA  interface  and  ( 
(enable/disable  EDA  ontput( 
(commands.  Test  EHA  write ( 
(protect  and  the  fault  ( 
(register  indicator  Bit  1. ( 
(Use  external  test  hard-  ( 
(ware  to  initiate  EM  A  I 
(read/write.  ( 

Modify  fcr  MIl-STD-1750 
Verification  as 
necessary 

1 "  .  '  "  " 

ITrigcer  GO 

1  Indicator 

1 

1 

(Test  for  output  discrete  [ 
(and  40  ms  timeout.  ( 
(Use  oscilloscope  to  ( 
(measure  timeout.  ( 

Modify  for  MIL-SID-1750 
Verification  as 
necessary 

1 

I Illecal 

1 

1 

1 

1 

(Check  fault  register  ( 
(setting.  1 
(Execute  illegal  output  ( 
(command  and  test  for  | 
(expected  results.  j 

Modify  for  MIL-STD-1750 
Verification  as 
necessary 
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Table  8-1. 

IIIL-STD-1750  Applicability  of 

ATP  Subtests  (cont) 

1  AN/AYK-15A 

1  Subtest 

1  1 

1  Purpose  1 

BIL-STI-1750 

Applicability 

IPower  Off 

1 

1 

1 

1 

1 

1 

1  Verify  power-down  inter-  | 
irupt;  100  microsecond  | 

I  bold  time.  1 
llnitiate  power  down;  1 
(execute  consecutive  | 
(memory  stored  after  power | 
(on.  1 

Modify  for  MIL-STD-1750 
Verification  as 
necessary 

1  Power  On 

1 

1 

! 

1 

1 

1 

1 

1 

1 

(Verify  power-on  discrete; ( 
(execution  from  bootstrap  ( 
(HOM;  power-on  reset  ( 
(discrete,  power-on  BCE  ( 
(quiescent  state.  ( 
(Initiate  power-on;  ( 
(execute  from  FOB  to  r€ad/( 
(reset  power-on  discrete;  ( 
(verify  BCB  state  by  r€ad-( 
(ing  control  registers.  ( 

Modify  for  MIL-STD-1750 
Verification  as 
necessary 

Optional  in  MIl-STD-1750 

8.2.2  Phase  2  z  Random  Instruction  Generator  Develocment 


Ic  complete  tne  second  phase  of  testing,  a  Eandcm  Instruction 
Generator  Program,  and  related  control  programs  must  be  developed,  and 
the  existing  illL-STD- 1 750  simulator  modified.  The  follcwing  is  a  list 
of  characteristics  of  this  approach; 

1.  The  fiandom  Instruction  Generator  must  be  designed  to  accept, 
as  user  input,  a  seed,  from  vhich  the  random  sequence  of 
instructions  and  data  can  be  generated. 

2.  The  HIL-STD-1750  simulator  must  be  modified  to  facilitate 
the  inspection  and  modification  of  all  registers  and  cote 
locations  as  veil  as  start  and  stop  control.  It  must  be 
hosted  on  the  VAX  11/780. 

3.  The  following  describes  this  Fandom  process. 
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6.2. 2.1  2andoo  Program  Description 


It  is  possible  to  split  the  cverall  program  into  separate  structures 
cr  routines.  This  is  done  in  order  to  develop  detailed  operations 
based  upon  individual  module  function.  Ihe  structure  is  depicted 
(Figure  S-l)  with  names  associated  with  individual  functions.  This 
reflects  the  overall  concept  only,  since  further  detailing  obviously 
requires  a  further  refinement  of  the  functicrs- 


Figure  8-1.  Program  Hierarchy 


8.2.2. 1.1  Program  Description.  In  reference  to  Figure  8-1,  a  brief 
description  of  each  block  can  provide  a  better  understanding  of  the 
cverall  concept: 

1.  ''KAIN'*  controls  the  program  cycling;  the  generation, 
simulation  and  execution  of  test  cases. 

2.  Ihe  "PAPAHETEBS"  subroutine  provides  parameter  values  for 
use  by  the  remainder  of  the  test  program. 

3.  Ihe  "BUILD"  subroutine  creates  the  operation  code  pool  from 
which  OP  CODES  are  taken  for  the  pseudo-random  instruction 
stream. 
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U.  The  "GENEBATE"  subroatine  builds  the  instruction  stceas  by 
randoBly  selecting  an  OP  CODE  and  then  building  the  operands 
for  each  instruction. 

5.  The  ••TEST"  subroutine  controls  test  case  siaulation  and 
execution  until  an  error  is  detected  or  the  end  of  the 
instruction  stream  is  encountered. 

6.  Ihe  "SiaULATOB"  subroutine  sioulates  each  instruction  found 
within  the  pseudo-randcn  instruction  streaa. 

7.  The  "EXECOTE"  subroutine  causes  the  execution  of  the 
instruction  strean  by  the  processor. 

8.  The  "CCtlPAfiE”  subroutine  ccnpares  the  expected  results  frcis 
the  SiDulator  subroutine  with  the  actual  results  from  the 
Execute  subroutine  to  deteraine  if  an  error  has  been 
generated. 

9.  The  ••OOTPCIT"  subroutine  produces  all  hardcopy  resulting  from 
test  piogran  esecution. 


8.2. 2. 1.2  "JUU"  Routine.  The  "Bain"  routine  is  the  topaost  module 
in  the  series  which  comprises  the  Pandca  Instruction  Test  Program. 
Its  function  is  to  control  the  execution  of  the  program.  The 
following  operations  take  place  within  this  routine. 

•  A  Program  Control  Area  (PCA)  provides  the  area  for  parameter 
input  from  the  operator  and  ether  system  supiplied  values. 

•  Restart  rs  checked  by  a  test  of  the  restart  bit  in  tne  PCA. 
If  restart  has  occurred,  control  is  passed  tc  Output  routine 
to  provide  tre  operator  with  restart  irfcrmaticn. 

•  The  machine  architecture  is  determined  by  testing  the 
appropriate  field  in  the  PCA.  Cnee  the  architecture  is 
determined,  the  instruction  table  for  that  architecture  is 
brought  into  the  system,  if  not  currently  leaded  as  in  the 
case  of  restarts. 

•  A  storage  area  is  maintained  for  module  communications. 
This  communications  ccrtrcl  block  contains  pointers, 
structures,  and  flags  for  usage  by  other  modules. 

•  An  initial  machine  state  is  established. 

•  Ihe  "BAIN"  program  loops,  generating  test  cases  until  an 
operator  initiated  stop  is  encountered. 

Control  is  passed  in  the  following  sequence  to  subroutines  which  make 
cc  the  "BAIN"  program  loop. 
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•  Ihe  "PARAMETEFS"  routine:  Provides  parameter  values  for  the 
remainder  of  the  test  program. 

•  The  "BUILD"  routine:  Generates  an  OP  CODE  pool  for  use  in 

instruction  stream  generation. 

•  The  "GENERATE"  routine:  Generates  a  pseudo-random 

instruction  stream. 

•  Ihe  "TEST"  routine:  This  routine  causes  all  the  testing  by 
passing  control  to: 

-  The  "SIKOLATCE"  routine:  Simulates  the  instruction 

stream. 

-  The  "EXECOTE"  routine:  Executes  the  instruction 

stream. 

-  The  "CCHEASE"  routine:  Determines  if  an  error  has 

occurred. 

-  The  "OGTPOT"  routine:  Provides  hardcopy  output. 


6.2.2. 1.3  "PABAaETERS"  Routine.  Parameters,  after  being  validity 
decked  by  the  parameter  processor  of  the  "BAIN"  program,  become 
available  to  the  "BAIN"  program  when  a  request  is  made  for  operator 
input.  The  function  of  this  routine  is  to  provide  parameter  values 
for  use  by  the  remainder  of  the  "BAIN"  pregram.  Those  values  may  be  a 
randemized  selection  of  the  operator  input  or  a  randemized  selection 
of  the  available  options.  operator  input  is  verified  and  a 
randomizing  of  values  is  performed  for  each  pass  through  the  "BAIN" 
program.  The  operation  of  this  routine  follows: 

•  A  base  seed  for  random  generation  of  data  is  requested. 
Note  that  loading  this  base  seed  permits  a  cemputer  that  has 
previously  failed  certification  to  be  retested  using  the 
same  psuedo- random  test  sequence. 

•  A  check  is  made  to  determine  if  operator  input  is  available. 

•  If  input  is  supplied,  the  buffers  size  parameter  (which  is 
used  to  track  storage  allocation  and  usage)  is  checked  for 
values  within  sp«^^  fied  ranges. 

If  not  in  range,  a  request  is  reissued  with  a  message 
indicating  the  operator  error. 

•  If  buffer  size  differs  from  its  previous  value,  then  storage 
is  released  and  the  new  storage  size  is  used. 

•  OP  and  NOP  parameters  which  ccntrol  execution  are  checked 
for  valid  OP  CODES,  and  if  the  sane  OP  CODE  is  found  in  both 
parameters,  the  NOP  parameter  overrides  the  CP  value.  The 
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OP  and  NOP  parameters  are  used  to  define  which  of  the 
instructions  have  been  implemented  in  the  computer  under 
test. 


8.2.2. 1.4  "BUILD"  Routine.  The  function  of  this  routine  is  to 
provide  a  table  of  valid  OP  CODES  for  use  by  the  remainder  of  the 
program.  By  generating  a  separate  table,  in  lieu  of  the  architecture 
table,  there  is  a  reduction  in  the  amount  of  time  spent  in 
verification  of  OP  CODES  found  in  the  random  instruction  stream.  This 
CP  CODE  pod  is  formulated  in  the  fcllcwing  manner. 

•  On  entry  to  the  routine,  pointers  are  available  to  the  OP 
parameter  OP  CODES,  NOP  parameter  CP  CODES,  the  architecture 
table,  and  to  the  storage  area  which  cbntains  the  OP  CODE 
pool. 

•  Depending  on  the  contents  of  the  OP  or  NOP  parameters,  all 
OP  CODES  for  the  randomly  selected  machine  features  are 
processed. 

•  If  OP  is  specified,  only  those  CP  CODES  are  transferred  to 
the  OP  CODE  pod. 

•  If  NOP  is  specified,  all  OP  CCDES  with  the  exception  of 
those  specified  are  transferred  tc  the  OP  CODE  pool. 

•  This  routine  loops  until  all  CP  CODES  arc  processed. 


£.2.2. 1.5  "GENERATE"  Routine.  The  Generate  routine  is  responsible 
fcr  providing  a  random  instruction  stream  with  the  appropriate  values 
established  in  the  corresponding  contrd  registers,  general  purpose 
registers,  floating  point  registers  and  data  areas.  In  order  that  the 
test  program  is  able  to  distinguish  between  processor  execution 
results  and  simulation  results,  two  copies  of  the  randomly  generated 
instruction  stream  are  available.  Therefore,  one  stream  is  used  as  a 
reference,  one  stream  for  simulation  and  one  stream  for  processor 
execution.  The  function  of  this  module  is  accomplished  by: 


• 

A  random  selection  of  CF 
previously  built. 

CODES 

from 

the  OP  CODE 

pool 

• 

As  each  OP  CODE  is  placed 

into 

the 

random  stream. 

the 

operands  for  that  particular  CP  CODE  are  formed. 


•  As  operands  are  placed  into  the  random  stream,  any  register 
or  data  area  associated  with  the  operand  is  provided. 

•  Interrupts  are  forced  into  the  stream. 

•  Two  copies  of  the  random  stream  and  data  areas  are  made 
available. 
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8.2.2.  1.6  “TEST**  Eoatine.  The  function  of  this  routine  is  to 
continue  test  case  execution  until  either  an  ending  sequence  for  the 
instruction  stream  or  an  error  is  detected.  This  routine  controls  the 
seauence  of  execution  of  the  "TEST",  "EXECDTE",  "CCHPAEE”  and  "OUTPUT" 
ccutines.  The  sequence  of  events  through  this  module  is: 

•  Preparation  of  pointers  followed  by  the  sinulation  of  the 
random  instruction  stream. 

•  Provide  a  random  stream  for  execution  on  the  processor  and 

the  saving  of  the  results  cf  that  execution. 

•  Cause  a  comparison  of  the  results  of  the  simulation  to  the 

results  of  the  execution  tc  determine  the  extent  of  success 
of  processor  execution. 

•  Cause  a  hardcopy  of  stream  and/or  data  areas  tased  on 

operator  parameter  selecticn. 

•  Cause  an  attempt  to  isolate  on  a  detected  processor 

execution  error. 


8.2.2. 1.7  "S1MULAT0&"  Houtine.  This  routine  simulates  the 

instruction  provided  to  it  from  the  random  instruction  stream.  This 
is  accomplished  in  two  basic  steps:  (1)  a  simulation  of  instruction 
fetch  as  performed  by  the  processor  unit,  and  (2)  a  simulation  of  the 
instruction  utilizing  a  basic  set  of  instructions  for  simplicity.  A 
further  breakdown  of  events  fellows. 

•  The  instruction  length  code  and  address  of  the  next 

sequential  instruction  are  calculated  and  saved. 

•  The  architecture  table  entry  for  the  OP  CODE  of  this 

instruction  is  obtained  to  determine  the  various  types  of 
interrupts  which  can  occur  from  this  instruction. 

•  A  determination  is  made  as  tc  whether  this  instruction  is 
capable  of  causing  interrupt  to  occur. 

•  The  OP  CODE  is  checked  tc  determine  if  it  is  legal, 

privileged  or  illegal  with  corresponding  interrupt  if 
necessary. 

•  The  OP  CODE  format  is  used  to  determine  if  ether  interrupts 
can  occur  from  the  specification  cf  the  operands. 

•  If  no  interrupt  is  detected  tc  this  point,  the  instruction 
is  simulated. 


8.2.2. 1.8  "EX ECUTE"  Routine.  This  routine  provides  the  randomly 
generated  instruction  stream  to  the  processor  for  execution.  The 
system  environment  is  maintained  by  trapping  the  state  of  the  machine 
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piior  to  Instruction  stream  execution  and  restoring  that  state  upon 
return  from  execution.  The  events  which  cause  this  transition  are: 

•  The  current  state  of  the  machine  (that  is;  the  contents  of 
general  purpose,  floating  point  and  control  registers  as 
well  as  the  system  status)  is  chtained  and  saved. 

•  The  register  contents  required  for  stream  execution  replace 
the  general  purpose  and  floating  point  registers. 

•  Any  control  registers  are  altered  tc  reflect  the 

requirements  of  the  stream  to  he  executed. 

•  The  Machine  State  (Instruction  Counter,  Status  Word,  and 
System  Interrupt  Mask)  to  cause  the  random  instruction 
stream  to  start  execution  on  the  processor  is  established. 


•  An  "in-the-stream"  indicator  is  set  to  identify  the  last 
location  of  execution. 

•  The  new  state  of  the  system  is  imposed  and  the  processor 
begins  to  execute  the  random  instruction  stream. 

Control  does  not  return  until  the  execution  of  the  random  stream 
results  in  an  interrupt  and  that  interrupt  is  processed.  At  this 
time,  the  "in- the-stream”  indicator  is  reset  and  the  Exerciser  routine 
is  complete. 


8.2. 2. 1.9  •'COMFAEE”  Soutine.  This  routine  compares  the  results  of 
the  Simulator  routine  from  Simulating  instructions  and  the  results  of 
the  Execute  routine  from  processor  execution  to  determine  if  an  error 
has  occurred. 

In  general,  all  machine  state  and  data  areas  (including  registers) 
from  the  Simulator  and  EXECUTE  routines  should  match  at  the  time  this 
routine  receives  control.  There  may  exist  cases  which  are  special  and 
which  would  not  result  in  equality  between  all  fields;  they  are 
architectural  dependent  and  must  be  defined  "a  priori",  and  handled 
specially. 


8.2.2.1.10  "OUTPUT"  Routine.  Hardcopy  results  cf  the  test  cases  are 
provided  via  this  subroutine.  A  varying  arrcunt  of  printer  output  is 
available  based  on  operator  specified  parameters,  values  of  operands 
within  the  random  instruction  stream,  and  the  location  of  an  error  if 
cne  has  been  detected. 

Cn  CPU  restart,  the  following  informaticn  is  provided; 


A  display  of  the  program  control  area. 
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•  A  display  of  the  operator  selected  and  program  randomized 
parameters. 

•  A  display  of  the  OP  CODE  counters,  i.e.,  instructions  which 
have  been  executed. 

•  If  a  random  instruction  stream  exists,  then  a  display  on: 

-  The  random  instruction  stream  in  a  readable  format. 

-  The  interrupt  history,  or  the  results  of  the  acutal 
instruction  stream  execution  versus  the  results  of  the 
simulation  of  the  instruction  stream. 

The  control,  general  purpose,  and  floating  point 
registers  associated  with  the  stream. 

-  Any  data  area  which  is  referenced  by  the  stream. 

Irror  information  and/ct  trace  information  is  provided  dependent  on 
the  operator  selection  of  a  "PBINT"  parameter.  If  an  unexpected 

interrupt  or  a  data  error  occurred  and  a  printout  is  desired  then  the 
following  information  is  provided: 

•  The  program  history,  if  requested  via  a  •'HISTCBX“  parameter. 

•  The  operator  selected  and  program  randomized  parameters. 

•  The  interrupt  and  OP  CODE  counters  if  requested  via  "COUNT” 

parameter.  This  parameter  is  reset  by  the  program  and  must 
be  respecified  for  subsequent  counter  displays. 

•  The  random  instruction  stream  in  a  readable  format. 

•  The  interrupt  history  of  the  results  of  the  actual 

instruction  stream  execution  versus  the  results  of  the 
simulation  of  the  instruction  stream. 

•  The  control,  general  purpose  and  floating  point  registers 
associated  by  the  stream. 

•  Any  data  area  which  is  referenced  by  the  stream. 

•  A  trace  at  end  of  stream  does  not  include  a  display  of  the 
program  history. 

•  A  trace  after  interrupt  comparison  does  not  include  operator 
and  randomized  parameters,  or  the  interrupt  and  CP  CODE 
counters. 

•  A  data  error  causes  a  double  display  to  be  made.  One  area 
contains  the  error  while  the  second  area  is  the  reference. 
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8.2.2.1.11  Architecture  Table.  For  each  architecture  subset 
permitted  via  the  optional  MIL-STD- 1750  functions  cr  features,  a 
separate  architecture  table  is  provided.  This  table  is  a  variable 
length  table  which  contains  all  the  rules  cf  usage  of  each  OP  CODE 
(and  OP  CODE  extensions)  supported  by  that  particular  architecture. 
It  also  identifies  those  OP  CODES  (and  CP  CCDE  extensions)  which  may 
net  be  used  by  an  architecture.  (Note  that  the  command  word  from  the 
I/O  instruction  should  be  considered  as  an  extension  to  the  OF  CODE.) 

The  architecture  table  is  established  by  means  cf  a  scries  cf  macros. 
Once  complete,  the  table  is  presented  to  the  test  program  as  a 
directly  accessed  table. 


8.2.3  Costs 


The  following  resources  with  their  respective  costs  are  required  to 
inplement  the  Two  Level  MII-STD-1750  certification  capability  at 
SIAFAC.  Note  that  the  Non-Recurring  Start-Dp  cost  of  1933,000  and  the 
Recurring  cost  of  $306,000  sums  to  a  tctal  cost  of  $1,239,000  which  is 
less  than  the  sum  of  the  ATP  approach  and  Random  approach  alone 
(51,470,800  =  617,000  +  853,800)  because  seme  duplicaticn  of  effort  is 
eliminated. 


8-22 


e  176175A 


FINAL  EEPCEI 


February  29,  1960 


ITEM 

SCFTEAEE 

Development  Computer  System  Otilities 
Support  Software 

-  Cross  Assembler  for  MIl-STD- 175 C 

-  Linitage  Loader 

-  Simulator 

Modification  of  AN/AYK-15A  ATP 
Modification  of  MIL-STD-1750  Simulator 

Development 

-  Bootstrap  Program 

-  Random  Instruction  Generator 

-  Control  Program  on  Master 

-  Supervisor  Program  on  Slave  for  Eandom 

-  Source  Tape  Generator  Program 

-  I/C  Test  Programs 

E AEDNAEE 

Development  Computer 
Master  Computer 

HIL-STD-1553  Serial  Channel  Interface 
ES-232  Serial  Channel  Interface 

CTREE 

Test  Plan  Document 


Non-Recurring 
Start-Up  Cost 
(Mar  Years) 


N.C. 


N.  C. 
N.C. 
N.C. 

3.  1 

2.0 


0.3 

3.0 

1.8 

l.u 

0.3 

0.5 


N.C. 

N.C. 


N.C. 


0.3 


SUBTOTAL 

System  Integration  Factor  (5K) 


12.7  man  years 
0.63 


TOTAL 


13.33  man  years/ 
$933,000 
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Ike  Becurriag  cost  for  30  conputers  is  5306,000  =  ($10. 2K  *  30). 
figure  is  derived  from  the  following  information: 

Cost 

fiecurring  Cost  Item/Compoter  (K  Dollars) 


Hardware  Maintenance  0 

Software  Sustainence 

-  ATP  4.14 

Bandom  2.7 

Personnel 

Coverage  to  Initialize,  Observe,  and  2.8 

Analyze  Results  (2  people  for  1  week) 

-  Technician  to  Supervise  Integration  of  0.28 

I/O  Interface 


Other 

-  Test  Plan  to  Vendor  with  Verification  0.28 

Source 


TOTAL  10.20 


This 
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6.3  CEBTIFICATION  SCENARIO 


After  all  phases  of  the  MIL-SID-n50  Certification  Procedure  have  been 
designed,  implemented,  documented  and  debugged,  the  following  series 
of  events  will  take  place  concerning  a  vendor  certification: 

1.  A  vendor  contacts  SEAFAC  personnel,  citing  the  need  for  a 
certif ication- 

2.  SEAFAC  makes  available  to  the  vendor,  what  reguirements  must 
be  met  to  complete  the  certification  (software,  GSE,  and 
I/O) ,  providing  source  modules  of  the  static  verification 
program  used  in  the  first  phase  of  testing  in  the  form  of  a 
Certification  Test  Interface  Document  (see  Appendii  E)  . 

3.  Time  is  allocated  for  the  certification  process  to  take 
place. 

4.  The  vendor  arrives  on  site  with  the  ail-STr-1750  computer 

and  related  Ground  Support  Equipment,  and  is  given  time  to 

shakedown  and  cable-up  his  hardware. 

6.  Ihe  first  phase  of  verification  testing  is  conducted  under 
supervision  of  SEAFAC  personnel  and  vendor  representatives. 
Any  discrepancies  are  so  noted,  all  results  are  documented. 

6.  The  second  phase  of  verification  testing  is  initiated  and 

left  to  execute  for  a  predetermined  length  of  time.  The 
length  of  time  is  proportional  to  the  speed  of  the  unit 
under  test  and  the  number  of  test  cases  desired  to  be  run. 

7.  After  a  thorough  review  of  all  test  results,  an  official 

statement  of  compliance  or  non-compliance  is  made  by  SEAFAC 
concerning  the  vendor's  MIL-STr-1750  computer. 


8.4  IMPACTS  TO  MIL-STD- 1750 


The  recommended  approaches  do  not  required  any  special  features  to  be 
added  to  MIL-STD-I 750  because  of  verification  tests. 


8.4.1  WIL-STD-1750  ARCHIT ECTDBE  CONTROL 


A  complete,  detailed,  unambiguous  specification  of  any  computer 
architecture  (like  MIL-STD- 1750)  is  essential  to  a  certification 
effort  in  order  to  ensure  that  a  compatible  verification  is  possible 
for  all  vendor  implementations. 
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He  architectural  specification  (MIl-STD- 1750)  itself  is  not  iomutable 
(fixed)  for  all  time;  rather,  ccrrections,  modifications  and 
extensions  will  need  to  be  made  over  time.  Thus,  a  change  mechanism 
needs  to  be  provided  foi  HIL-STD-1750.  Control  of  potential  changes 
is  critical  by  this  change  mechanism  and  must  be  predicated  upon  a 
ctcund  rule  that  upward  compatibility  is  required.  That  is,  a  program 
written  for  a  HIL-STD-1750  implementation  will  run  and  give  the  same 
time  independent  and  implementation  independent  results  on  an 
implementation  based  on  MIL-STD-175CA.  (Of  course,  this  assumes 
similar  resources  for  both  computers  such  as  32K  storage  on  both 
machines. ) 

The  upward  compatibility  enables  the  verification  program  to  migrate 
successfully  from  an  implementation  based  on  MIL-STD-1750  to  an 
implementation  based  on  MIL- STD- 17 50 A.  (Of  course,  the  verification 
program  would  need  to  be  extended;  but  deletions  or  changes  would  not 
be  required.) 

Thus,  it  is  required  that  different  release  levels  cf  the  verification 
program  would  need  to  be  maintained  with  at  least  one  release  level 
associated  with  each  MIL-STD-1750  release  level- 


8.U.2  SDBSETS  OF  HIL-STE-1750 


The  issue  of  defining  architectural  subsets  has  an  important  design 
impact  on  the  certification  process’s  verification  approach. 

Subset  specification  guidelines  cr  rules  must  be  provided  by  the 
PI1-STD-175C  specification  and  certification  testing  must  conform  to 
these  rules.  Since  the  existence  of  subsets  will  require  special 
treatment  in  the  verification  approach  (as  well  as  either  multiple 
assemblers  or  a  mote  complex  assembler)  with  subsetting  capability, 
subsetting  should  be  examined  carefully.  The  follcwing  is  a  candidate 
process  for  providing  member  specif ica tier; 
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BULE5  CP3ION 

Fixed  Point 

[  ]  16-Eit  Arithmetic 

[  ]  32-Eit  Arithmetic 

[  ]  16-Eit  and  32-Eit  Arithmetic 

Pleating  Point  Arithmetic 

[  ]  Short  (24/8) 

[  ]  Shert  and  Long 

[  ]  Expanded  Addressing 

[  ]  MIL-STE-1750  I/O  Protocol 

-  EIL-STD-15E3  Subset 

[  ]  Protection 

-  Kemory 

Superviscr/Protlem  States 
[  ]  Start-Dp  BOM 

[  ]  DMA 

Thus,  many  subsets  of  the  architecture  would  be  peroitted  following 
this  set  or  rules.  The  simplest  member  would  be  characterized  as  a 
16-bit  fixed-pcint  processor;  the  other  extreme  would  comprise  the 
entire  architecture. 

Therefore,  multiple  architecture  subsets  reguire  special  design 
considerations  for  the  verification  program  in  order  that  it  properly 
tests  all  possible  machine  implementations. 

Parenthetically,  the  same  design  considerations  is  required  for 
support  software.  For  example,  the  compiler  may  need  to  flag  floating 
pcint  equations  as  illegal  for  the  simplest  member.  The  assembler  may 
likewise  need  to  flag  illegal  instructions  for  this  simplest  member. 


Must  Choose  One 


May  Choose  One 


May  Choose 
(options) 


8.5  IMPACTS  TO  SEAFAC 


The  impact  to  SEAFAC  due  to  implementing  the  MIL-STD-1750 
certification  capability  as  a  two  level  approach  focus  on  staffing  and 
hardware . 


8.5.1  Staffing 


Staffing  requirements  necessary  to  implement  (totally  within  SEAFAC 
without  any  subcontracted  work)  the  recommended  verification 
approaches  consist  of  the  following: 
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Ingineer  to  iiplement  the  I/O  interfaces 

lead  Programise  with  an  architectural  background 

Fupport  prograamers  (5) 

The  staffing  required  to  sustain  the  reccnniendation  follows; 

Engineer/technician  to  supervise  the  1/C  integration 

PrograEBet/technician  to  initialize,  invoke,  and  observe 
verification  prograo  execution 

Certification  Coordinator 

Programmer  to  maintain  the  vetificaticn  programs. 

6.5.2  Physical  Besources 

The  physical  resources  necessary  to  implement  and  sustain 
recommended  verification  approaches  consists  of  the  following; 

Master  Computer  with  Associated  Peripherals 
Auxiliary  Storage  (disks) 

Magnetic  Tape  Drives 
Printer 

Timesharing  Operating  System 

HIL-STD-1553  Interface  from  the  Master  Computer 
PS-232  Interface  from  the  Master  Cciputer 
MIL-SID-1750  Support  Software 
Cross  Assembler 
Linkage  Editor 
Simulator 
Library 
Utilities 
Power/Lab  Space 


1980 


the 


t  he 
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9 . C  EPILOGUE 


lEH  is  fully  confident  in  the  results  of  this  study  as  documented  in 
this  repcrt;  however,  it  is  important  tc  recognize  that  these  results 
are  based  upon  the  set  of  ground  rules  (developed  with  Air  Force 
guidance)  which  are  described  in  Section  2.  Different  results  could 
be  obtained  if  these  ground  rules  were  changed  or  modified;  therefore, 
it  is  important  to  take  these  results  in  the  context  of  the  ground 
rules.  However,  enough  information  has  teen  provided  in  this  document 
tc  permit  re-evaluation  of  the  recommendation  if  the  ground  rules 
change . 


9.1  OESIEVATICNS/COMMENIS 


luring  the  study  phase  of  this  contract,  many  interesting  guestions 
were  discussed  by  the  IBM  team  members.  This  section  cowers  those 
guestions  with  our  responses. 

"Since  either  the  dIL-STC-1750  specification  will  mature  over  time  or 
else  the  verification  program  will  mature  over  time,  what  happens  to  a 
previously  certified  computer?" 

Betesting  needs  to  be  considered.  A  reasonable  scenario  for 
retesting  would  be  to  recall  all  certified  computers  for  retest 
whenever  the  verification  test  is  updated.  Any  discrepancies 
found  would  be  recorded  and  published  so  that  users  would  be 
aware  of  the  limitations  of  these  existing  computers  vis  a  vis 
the  latest  level  of  standard  or  verificaticn  method. 


"Nhat  advantages  or  disadvantages  arise  from  supplying  vendors  with 
the  verification  programs?" 


The  advantage  to  the  vendor  is  that  each  implementation  could  be 
pretested  prior  to  the  certif icaticn  process  at  SEkF^C.  This 
could  make  the  certification  process  less  painful  to  the  vendor. 
The  advantage  to  the  Air  Force  is  that  more  mature  hardware 
implementations  would  be  submitted  for  certification  testing.  An 
expressed  concern  is  that  a  vendor  would  "hand  tune"  its 
implementation  to  only  consider  the  tests  in  the  verification 
program.  If  the  verification  program  is  of  high  quality,  this 
should  be  an  advantage,  not  a  ccncern.  Furthermore,  IBM  believes 
the  two  step  certification  testing  process  with  the  Random 
approach  as  the  second  step  eliminates  this  concern. 
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"For  the  Hanual  Test  Configuration,  isn't  it  improper  for  the  load 
tapes  to  be  generated  by  the  vendor?" 


In  the  Manual  Test  Configuration,  the  study  assumed  that  the 
vendor  would  generate  the  load  tapes  for  use  during  the 
certification  process.  It  was  assumed  that  keeping  the  load  tape 
for  future  reference  would  provide  adequate  control  and 
protection  for  the  Air  Force.  It  has  been  suggested  that  this  is 
an  improper  approach  since  merely  having  the  means  to  establish 
the  validity  of  the  test  after  having  conferred  certification  xs 
not  sufficient  in  light  of  the  cost,  schedule,  and  competitive 
damages  that  could  occur.  This  is  acknowledged  and  is  further 
rationale  for  not  recommending  the  Manual  Test  Configuration.  Cf 
course,  other  means  cf  providing  the  desired  level  cf  control  can 
be  envisioned  such  as,  requiring  a  standard  load  capability  for 
all  vendors.  However,  these  alternative  methods  all  have  added 
costs  associated  with  them  and  further  support  the 
recommendations  of  the  study. 


"Should  a  minimum  main  storage  si2e  be  required  for  the  certification 
process?" 


Yes,  a  32K  minimum  storage  size  is  required.  However,  some 
systems  might  specify  less  than  32K  storage.  For  these  systems, 
it  would  be  permitted  to  let  the  hardware  engineers  satisfy  the 
32K  storage  requirement  by  previding  a  memory  expansion  connector 
(which  may  be  inside  the  box)  tc  provide  an  external  memory 
extension  up  to  32K.  This  expansicr  could  be  of  commercial 
quality. 


"Should  a  standard  I/O  channel  like  HIl-STr-1553  or  RS-232  be  required 
for  certification?" 


A  standard  MIL-STE-1553  interface  is  required.  For  systems 
without  MIL-STD-1  553 ,  the  hardware  engineers  again  could  provide 
this  feature  by  previding  an  I/O  ccnnector  (which  may  be  inside 
the  box)  which  connects  to  an  external  box  providing  the 
MIL-STD-1553  function;  or  the  hardware  engineer  could  provide  an 
FS-232  connector  in  the  box.  The  external  I/O  tex  could  be  of 
commercial  quality. 
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ffluch  eccoc  reporting  should  he  provided  to  the  vendor  unen  a 
rrctlem  is  encountered  during  certi f icat icn  testing?” 


The  state  ot  the  aachine  at  the  start  of  a  test  case,  that 
specific  test  case  data,  the  results  of  that  test  case  after 
execution  upon  the  machine  under  test,  and  the  expected  results 
(either  predetermined  or  simulated)  should  be  provided  for  each 
test  failed  during  the  certif ication  process.  However,  it  is  not 
considered  the  province  of  the  Air  Force  to  provide  diagnostic 
information. 


"Shouldn’t  future  research  be  considered  for  either  the  Analytical 
Approach  or  the  Computer  Hardware  Cescription  Language  (ISPS) 
approach?" 


Further  consideration  should  be  given  to  describing  the 
HTL-STD-175Q  architecture  in  these  approaches  in  order  to  mature 
the  architecture  specification  through  a  thorough  simulation 
effort  and  to  increase  the  confidence  in  this  architecture. 


"Are  any  additional  functions  required  by  the  Air  Force  to  further  the 
standardization  effort?” 


Perhaps  the  Air  Force  could  establish  a  libfary  of  operational 
software  functions  li)ce  SlUE,  CCSINE,  AECTANGENI,  NAVIGATION, 
K ALMAN^FILTEfi ,  etc.  Perhaps  the  Air  Force  cculd  provide  funds 
for  membership  participation  in  the  HIL-STE-1750  Users  Group. 
Perhaps  the  Air  Force  could  put  an  organization  in  place  to 
control,  distribute,  and  maintain  Support  Software  like  High 
Order  Languages,  Assembler,  Simulator,  Linkage  Editor,  debugging 
tools,  etc. 


"How  does  one  determine  when  to  stop  the  Eandcm  verification 
approach?" 

A  statistical  analysis  of  this  problem  has  been  non-productive; 
engineering  judgement  has  to  be  used  to  determine  when  to  stop 
the  Bandom  program. 


ei76175» 


FINAL  FEPCPI 


Fekruary  29,  1980 


"Doesn't  connecting  to  the  VAI  11/780  strain  that  resource  ty  being 
dedicated  to  the  certification  process?" 


The  VAX  11/780  provides  a  irulti-prograBBable  capability; 
therefore,  the  verification  progras  on  the  VAX  11/780  could  run 
concurrent  with  other  applications. 


"Nhen  using  a  SandoB  verification  prograB,  how  does  a  test  seguence 
cet  repeated  for  a  coaputer  which  is  resukaitted  for  certification 
after  previously  failing  the  Fandcn  tests?" 


The  Bandoa  verification  progran  should  request  the  base  seed  at 
its  start.  This  permits  the  Generator  Procraa  the  same 
pseudo-random  sequence  of  instructions  based  upon  the 
initialization  seed  used  when  the  computer  first  failed. 


"hew  many  test  cases  are  required  per  an  average  instruction  to  test 
the  architecture  without  hardware  dependencies  being  krewn?" 


Architectural  verification  programs  average  25  tests  per 
instruction  in  order  to  thoroughly  test  the  architecture  without 
J  priori  knowledge  of  implementation  details. 


"^hat  testing  philosophy  should  be  used  regarding  exceptional  (error) 
conditions?"  (e.g..  Store  Multiple  (STM)  crossing  page  protection 
toundaries,  an  unnormalized  floating  pcint  operand.) 


Not  only  should  testing  for  normal  conditions  occur,  but  even 
more  important  is  the  testing  of  the  exceptional  (error) 
conditions  and  the  status  cf  the  machine  upon  completion  of  the 
error  event.  e. g. ,  a  Store  Multiple  crossing  page  boundaries 
from  an  unprotected  page  into  a  protected  page  should  terminate 
the  same  way  on  any  machine  being  tested;  that  is,  STM  should 
terminate  without  storing  into  the  protected  page  and  this  should 
be  so  tested. 


"Nhat  testing  philosophy  should  be  used  regarding  "Bequired", 
"Cpticnal",  "Beserved"  and  "Spare"  features  in  MIL-STD-1750?" 


"Required",  "Optional",  and  "Beserved"  define  architectural 
features  which  must  be  tested;  "Spare"  is  not  defined  and  cannot 
be  tested. 
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"Phat  testing  pnilosophy  should  be  applied  to  ^glied  conditions  in 
the  ?.TL-STL-  1750  architecture?" 


An  example  of  an  implied  condition  is  storing  ocly  the  IS,  SV , 
and  IC  into  main  storage  when  an  interrupt  occurs.  In  this  case, 
the  adjacent  locations  should  also  he  tested  to  guarantee  they 
did  not  change  (i.e. ,  defining  a  four  word  FSW  is  illegal, 
therefore,  implied  conditions  should  also  be  tested. 
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1C.0  APPENDIX  A  -  SEAPAC  FACILITIES 


The  System  Engineering  Avicnics  Facility  (SEAFAC)  is  an  avionic 
hct-hench  and  test  facility  within  Avicnics  Engineering  Directorate 
(ENA)  Aeronautical  Systems  Division  (AS!)  of  the  Air  Force  at  WPAFB. 
Originally  organized  to  analyze,  design,  simulate,  and  evaluate 
avionic  equipment  in  a  total  system  context,  the  facility  has  been 
tasked  to  play  a  key  role  in  the  standardization  effort  of  the  Air 
Force.  Through  SEAFAC,  the  capability  to  aid  users  in  the  application 
cf  standards  and  to  verify  ccmpliance  with  the  standards  is  being 
developed  on  an  evolving  basis  to  meet  the  requirements  of  tne  Air 
Force. 

kith  respect  to  stanoards  verification,  SEAFAC  has  been  the  principal 
agency  for  the  ClIL-STD-1553  bus  effort.  Three  generations  of  bus 
testers  have  been  designed  to  provide  both  in-house  and  on-site 
certification  and  system  engineering  of  NIL-STD- 1 55 3  equipment.  A 
number  of  projects  have  been  addressed  in  support  of  Program  Office 
activities  which  has  expanded  the  role  cf  SEAFAC  in  providing  benefits 
and  capabilities  to  ASD- 

The  resources  available  at  SEAFAC  consists  of: 

1.  Physical  Plant  -  Facilities  for  administrative  support  and 
capabilities  to  accommodate  hot-bench  system  ccnxigurations, 
hardware  design  and  fabrication,  and  testing  and  evaluation 
are  available. 

2.  Electronic  Hardware  -  A  variety  cf  hardware  resources  exist, 
including: 

a.  PDP-11/55  computer  system  with  links  to  a  aiL-STD-1553 
data  bus.  Mass  memory  is  provided  by  two  SK-05  disc 
drives  (each  containing  2.5  megabytes),  two  RK-06  disc 
drives  (each  containing  13.9  megabytes),  a  cuai  floppy 
disc  drive,  and  a  112K  word  memory. 

b.  PCP-11/34fl  with  32  kilcwcrds  of  main  memory  and  with 
custom  designed  bus  interface  unit  for  bus  controller 
or  remote  terminal  operation  on  a  MIL-STC-15h3  bus. 

c.  VAX  11/780  computer  system  with  1  million  bytes  of 
memory,  two  Et!03  disc  drives  (each  containing  64 
meganytes'  ,  2  switchatle  shared  RK-06  disc  drives. 

d.  Microcomputer  develctment  system  (e.g.,  IMP16,  SILENT 
700,  8080). 

e.  Remote  terminals  for  access  to  ASC  computer  center 
containing  a  CYEER  175  and  System  360/37C  emulator. 
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f.  MinicoBFUter  (CP-16A  and  AYK-8) . 

g.  Test  equipment  including  iteters,  scopes,  generators, 
logic  analyzers,  custom  designed  multiplex  data  Itus 
testers,  and  MIl-STE-1553  verification  equipment. 

3-  Software  -  Support  software  for  the  PDP-11/5E  consists  of 
the  BSX-IIM  operating  system.  Macro- 11  assembler,  and 
FOBTBAN  IV  -  PIOS  ccmpiler. 

U.  Personnel  -  The  SEAFAC  personnel  contingent  is  comprised  of 
hardware  engineers,  software  programmers,  and  technicians 
with  experience  in  configuring  and  implementing  hot-bench 
avionic  system  setups.  This  branch  of  the  Aeronautical 
Systems  Division  is  broken  into  three  working  groups; 
Hardware,  Software  and  Multiplex.  The  staffing  level  for 
each  of  these  groups  is  as  follows: 

Hardware 

Engineers  6 

Technicians  3 

Software 

Programmers  3 

Technicians  2 

Multiplex 

Engineers  U 

Technicians  1 
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This  section  describes  in  detail  the  methcd  for 
runber  of  architectural  discrepancies.  As 
Engineering  Changes  (HCs)  are  a  measure  of 
architectural  discrepa ncies  which  have  been  foun 
machine  design.  It  is  desirable  tc  know 
architectural  discrepancies  which  remain  in  a  m 
sell-off,  i.e.,  those  already  found  plus 
ur  d iscovered- 


projecting  the  total 
discussed  previously, 
the  total  number  of 
d  to  date  for  a  given 
the  total  number  of 
achine  at  the  time  of 
these  which  remain 


t  is  expected  that  the  rate  of  discovery  of  arcnitecturai 
iscrepancies  will  be  an  exponentially  decreasing  function  of  time, 
ritial  use  of  the  machine  will  be  for  the  development  and  validation 
f  operational  software.  This  new  software  will  exercise  the  machine 
n  new  and  previously  untested  ways.  It  is  expected  that  most 
iscrepancies  will  be  discovered  early  in  this  usage.  As  the 
pplication  programs  become  more  developed,  fewer  and  fewer  new 
xercises  of  the  machine  are  performed,  and  the  rate  of  error 
iscQvery  declines.  Finally,  after  the  applications  programs  are 
cmpletely  developed  and  in  use  for  some  time,  the  rate  of  discrepancy 
iscovery  approaches  zero.  This  may  be  plotted  as  shown  in  Figure 
0-1,  where 

f (t)  is  the  number  of  ECs  discovered  as  a  function  of  time. 

The  area  under  this  decreasing  extenential  curve  (i.e.,  the  integral) 
is  the  total  number  of  architectural  discrepancies  which  remain  in  the 
machine  after  seii-off,  which  is  the  desired  quantity. 

The  curve  is  of  the  form 


-bt 

f(t)  =  ae 


(1) 


Tc  calculate  the  area  under  the  curve,  let  C  egual  the  total  number  of 
architectural  discrepancies.  Then 
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2  0.1.1  The  Decreasing  Exponential  fiet bod 


In  order  to  compute  C,  it  is  necessary  to  fit  a  curve  (with  the 
correct  values  of  a  and  b)  to  the  data,  f(t),  which  might  appear  as  is 
shcwr  in  Figure  20-2. 

This  may  he  accomplished  by  taking  the  natural  log  transform  of  the 
data,  thereby  making  it  linear  and  then  performing  linear  regression 
analysis  to  obtain  the  best  fitting  line.  This  results  in  a 
semi-logarithmic  curve  fit.  (This  will  be  called  the  decreasing 
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Figuce  20-2.  Expected  Data;  ECs  after  Sell-off  versus  lime 


earonential  method.)  The  transform  is 


A  -bt 

In  (f  (t) )  =  ln(ae  )  =  In  (a) -bt  (5) 


The  right-hand  side  of  this  equation  is  in  the  form  of  a  straight 
line,  for  tihich  the  general  expression  is  y  =  mx  +  b,  where  m  is  the 
slope  and  b  is  the  y-intercept.  Linear  regression  analysis  yields  the 
slope,  in  this  case  b,  and  the  intercept,  in  this  case  In  (a),  which 
are  the  desired  variables.  To  obtain  the  value  of  C,  the  value  or  b 
and  the  anti-log  of  In (a)  are  taken  and  applied  to  Equation  (4). 
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7be  slope  and  intercept  are  calculated  as  fellows: 


slope  =  b 


2  t  . 


^  t  *  f(t  ) 
i=  1  i  i 


-  t  *  f(t) 


intercept  =  In  (a)  =  f (t  )  -  b  t 


where  tj^  are  the  months  after  sell-off  and  f  (ti)  are  the  corresponding 
runber  of  ECs  of  architectural  relevance  for  those  months.  N  is  the 
tctal  number  of  data  points.  The  average  value  of  ?‘(t^),  represented 
by  t(tT)  ,  is 


A 

f(t  ) 
i 


N  A 

I  ) 

i=1  i 


The  average  value  of  represented  by  tj^,  is 


The  variance  of  the  t  values,  <rt  ,  is 
N  2 

I  t  _ 

2  i=  1  i  2 

t  N 


an  application  of  this  method,  the  data  from  a  Systeni/370  model  may 
he  evaluated.  (It  is  not  intended  that  the  results  he  evaluated  in 
this  section.  This  evaluation  appears  in  Section  7.  The  data  are 
hrcucht  here  for  the  purpose  of  illustration.)  The  data  is  shown  in 
Tahle  20-1  and  is  plotted  in  Figure  20-3.  The  natural  log  transform 
cf  the  data  is  taKen,  using  Equation  (5),  and  the  best  fit  line  is 
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found  using  Equations  (6) , (7) , (8) ,  (9) ,  and  (10).  These  are  plotted  in 
Eicure  20-4. 


Table  20-1.  ECs  of  Architectural  Belevance  Frcm  A  SysteiB/370  Hodel 
+ - + 


Month  (t) 


^CE  (f(t)) 


1 

1 

1 

4 

2 

1 

4 

3 

1 

3 

4 

1 

3 

5 

1 

5 

6 

1 

2 

7 

1 

5 

8 

1 

3 

9 

1 

1 

10 

1 

4 

1  1 

1 

1 

12 

1 

0 

13 

1 

2 

14 

1 

4 

15 

1 

1 

16 

1 

2 

1 

1  Total 

1 

44 

) 

I 
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Note  that  no  £Cs  were  found  in  month  twelve.  This  presents  a  problem 
since  In  (0)  equals  minus  infinity.  For  this  analysis,  the  twelfth 
mcnth  was  assumed  to  have  one  EC.  This  problem  will  te  discussed  in 
mere  detail  later. 

The  straight  line  found  is 

A 

In  (f(t))  =  1.451  -  0.0668  t  (11) 

A 

A 

This  yields  the  desired  curve,  f  (t) ,  by  taking  the  anti-log  of 
Fqoaticn  (11). 


A  A 

A  ln(f  (t)  )  1.451  -  O.C668t 

f  (t)  =  e  =  e 

-  0. oeeet 

=  4.267  e 


(12) 


This  is  plotted  in  Figure  20-5. 


ECs 

Found 


Figure  20-5.  ECs  of  Architectural  Relevance  for  System/370  Model 

versus  Time  and  Best  Fit  Curve 


A  total  of  44  ECs  were  found  by  the  end  of  the  16th  month  as  shown  in 
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Table  20-1.  The  total  projected  number  cf  architectural  discrepancies 
is  found  as  follows: 


a  4.267 

C  =  -  - - -  63.88,  or  approxiirately  64  (13) 

b  0.0668 


In  other  words,  20  additional  discrepancies  are  projected,  but  remain 
cndiscovered. 

As  mentioned  previously,  a  difficulty  with  this  decreasing  exponential 
method  is  its  inability  to  handle  data  points  of  value  zero,  since  the 
natural  log  of  zero  is  minus  infinity. 


20.1.2  The  Cumulative  Cat a  Approach 

An  alternative  approach  is  to  use  the  cumulative  function,  which  has 
crly  ncn-zero  values. 

The  cumulative  function,  denoted  by  g(t),  is  the  integral  of  the 
original  data  function,  f(t). 


a  -bt 

-  e  +  C  ,  or  using  Equation  (4) 

b  o 


-bt 

=  C  -  C  e  (14) 

o 


It  is  known  that  at  t  =  0,  no  ECs  have  been  found.  Therefore  g  (0)  =  0 
at  t  =  0.  Using  Equation  (14), 


E-fa 
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-h  X  0 

c{0)  =  C  -  C  e 

o 

0 

0  =  C  -  C  e 

o 


(15) 


c  =  c 

o 


The  resulting  form  of  the  cuEulative  function  is: 


-bt 

g(t)  =  C  -  C  e 


(16) 


This  is  plotted  in  Figure  20-6. 


Figure  20-6.  Cumulative  ECs  Over  Time 

The  asymptote,  C,  is  the  total  projected  number  of  architectural 
discrepancies. 

The  goal  once  again  is  tc  fit  the  best  curve  to  the  actual  data,  i.e, 
determine  the  best  values  c£  c  and  b.  An  example  of  vhat  the 
cumulative  data,  g(t) ,  might  look  like  is  shown  in  Figure  20-7. 
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Cumulative 

ECs 


Months  since  sell-off 

Figure  20-1.  Cumulative  ECs  Over  Time,  Data 


It  vouia  be  desirable  to  take  the  log  trarsform  of  tbe  data  (to  make 
it  linear),  fit  a  straight  line  by  linear  regression,  and  then  take 
the  anti-log  to  obtain  the  curve,  as  vas  dene  previously  with  f (t) . 
Ur  fortunately,  the  natural  log  of  C-Ce“^^  does  not  produce  a  straight 
line.  Note  tnat  the  derivative  (slope)  is 


a  -bt  1 

—  In  (C  -  C  e  )  - - 

dt  -bt 

C  -  C  e 


C  b  e 


1  -  e 


Since  the  slope  of  In  (C  -  C  e  )  is  net  a  straight  line,  but  is  a 
function  of  t  ,  linear  regression  analysis  cannot  be  used.  Therefore, 
the  follcHing  approach  to  fitting  the  best  curve  (cheesing  the  best 
values  of  C  and  b)  is  used. 
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First,  an  arbitrary  value  is  chosen  fcr  C  and  the  data,  g(t 
subtracted  from  it.  This  yields 

-bt 

h  (t)  =  C  -  g(t)  =  C  e 

which  is  plotted  in  Figure  20-8. 


Figure  20-8.  C  Minus  the  Cuirulative  EC  Function 


Next,  the  natural  log  is  taken. 
In  h(t)  =  In  C  -  bt 


E- 
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TFis  is  in  the  form  of  a  straight  lire,  fchich  is  plotted  in  Figure 
2C-9. 


In  [h  (t)  ] 

In  c 


Figure  20-9.  Log  Transform  of  Cumulative  Data 


Since  this  is  in  the  form  of  a  straight  line,  linear  regression 
analysis  may  be  applied  to  determine  the  lire  that  best  fits  the  dajta. 
Equations  (6),  (7),  (8),  (9),  and  (10)  are  applied,  escept  that  f  ("t^) 
is  reclaced  with  [In  "fi  (t^)  ]. 

The  slope,  t,  has  thereby  been  determined  and,  since  C  was  chosen,  the 
equation  for  the  curve  is  complete. 

The  difficulty  remains  that  C  was  chosen  arbitrarily.  The  solution  is 
to  choose  different  values  of  and  then  determine  the  goodness  of  fit 
between  the  resulting  curve,  g(t)  =  C  -  C  e"^^,  and  the  data,  §(t). 
The  tetter  the  fit,  the  better  the  estimate  of  C.  This  process  is 
then  iterated  until  the  best  estimate  cf  C  is  obtained. 

Two  different  measures  cf  the  degree  cf  fit  can  be  used:  the 
correlation  coefficient  (r)  of  the  line  calculated  by  regression 
analysis,  and  the  squared  error  of  the  calculated  cumulative  curve, 
^(t),  and  the  cumulative  data,  g  (t) . 

The  correlation  coefficient  is  a  measure  of  the  degree  of  association 
between  the  random  variables  (ln[h(t^)  ],  t^)  ....  {ln[h(tjj)J,  t^) .  It 
is  denoted  by  r  and  is  calculated  by  using  the  fcllcwing  expression: 
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r 


(  .L') 


!.'rte  that  ti.ii  apuixes  tt  tit  Itc  trar.  sfrrir  t  ♦!-  hitj,  ^  ,  t,. 

straight  iint.  The  i.ighet  the  r-valup,  the  the  tit. 

The  sauared  error  rs  a  measure  cf  the  fit  <-'t  t  .e  resulti;.^  curve  bitn 
the  data.  It  is  caiculatel  by  the  fcllcvirc  fcraula 


Squared  Error 


The  curve  shown  in  Figure  20-10  helps  in  visualizing  the  s^^uated  error 
concept. 


B-13 


FINAL  FEFCFI 


Fctruaiy  29,  1980 


Cumulative 

ECi 


rijure  ^0-lC.  '"cuarei?  Frier  T  ( j  reset,  t  at  i  c  r 

F'-r  «^dry  vc^ut  01  t  ,  tf€  :M:ffrercf  retkefr  tht  fitted  curve  and  the 
rata  i5  s^uaiec.  .ne.=;e  tetttJ  are  tFti.  sarrec.  The  tetter  the  fit  of 
c'jrve  t^  tt.e  s.ata,  tK  lower  tte  total  scuarec  errer. 

To  f  ummat  lit ,  ti.c  value  of  C,  wFicl  is  the  total  crc^ecteu  Lumber  of 
a  :  c  r.  i  t  e  r  t  u  ta  1  i  j.  ic  re  i  a  r  c  le  s  which  lemaii  ii,  the  machire  after 
sell-off.  Bay  i.t  estiirated  hy  use  of  ar  iterative  procedure.  This 
rirceiure  iLvoivfL  selectin';  various  arbitrary  values  cf  C  urtil  the 
best  es^-imat-  c.  i  o  is  found,  based  or.  a  maximuir  r  value  or  a  miLimum 
fouared  erre:  vaxut. 


*:  f 


I  A  ....  A I  a.)  N 


:  \  !  V  r  ACVTM 


V 


wa.f'.  li.>C..; 

aioViter^uicl  . 
'■ifer  t>aT  a 
e  r  t  if  ir  a  t  H-.I. 

fjirer^o'’  laBi't!  > 
r  r  e  '  ,  >  *  n  d  01  a  C’  n  .  Ii 


re  VI  ou  s 


tbe 


tro^ected 


total  luaiLer 


u  1  s.  t  e  (  a  nc  le  s  rust  be  roriralized  hy 
a  II  estimate  cf  Quality  be  ohtair.ed 
met  hoc.'.  That  is,  the  purpose  is 
a r rh 1  * ert ura  1  d i scr e ra ncie s  of 
)f  th<-  rrminal  size  exrected  for 


Bachir. e  sire  ic 
for  each  of  the 
tc  cempute  the 
a  verification 
Pll-STL- 1750. 


y 

.  I 


1  m  T  1  e .  t 
rraO  i  i.e  . 


•  1  !^c'^.nc  size  is  tbe  n  urn  her  of  catf.<=  used  ir, 

:t::.itily,  tli:  icrcrcs  tbe  cicicccde  anu  main 
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storage  of  the  machine,  either  of  hhich  (if  votking  incorrectly)  would 
result  in  an  architectural  discrepancy.  It  would  be  possible  to 
simply  sum  the  sizes  cf  the  logic,  microcode  and  main  storage. 
However,  it  is  expected  that  eaual  weighting  of  a  gate  cf  logic,  a  bit 
cf  microcode,  and  a  bit  main  storage  would  exaggerate  the  importance 
cf  main  storage.  Therefore,  each  of  the  three  components  is 
normalized  separately.  To  accomplish  this,  each  of  the  ECs  of 
architectural  relevance  uncovered  in  this  study  is  examined  to 
determine  whether  its  cause  is  logic-related,  microcode-related,  or 
main  storage-related. 

The  projected  numoer  of  architectural  discrepancies  remaining  in  the 
machine  after  the  application  of  Verification  Method  A  in  the  nominal 
size  MIL-STD-1 7 50  machine  (denoted  by  CISCPEP ( 1750)  )  is  the  sum  of 
three  components: 


DISCBEP(  /50)  =  Logic  DISCEEP  ( 1750)  ♦  Microcode  EISCEEP ( 1750) 

♦  Main  Storage  CISCPEP  ( 1750)  (B1) 


Assume  that  the  architectural  discrepancy  data  were  gathered  from 
Machine  B,  which  is  different  in  size  than  the  nominal  MlL-STD-1750 
machine.  The  first  ccmpcnent  in  the  above  equation,  the  number  of 
Icgic-related  architecture  discrepancies  expected  for  a  nominal  sized 
MIL-STD-1750  machine  after  the  use  of  Verification  Method  A,  becomes 

Logic  DISCBEP  (1750)  =  Logic  DISCBEE(E)  x 

machine  size 
normalization 

factor.  (B2) 

Where  Logic  DISCBEP (B)  refers  to  the  projected  number  cf  architectural 
discrepancies  associated  with  logic  found  in  machine  E. 

The  machine  size  normalization  factor  in  Equation  (B2)  is  simply 

machine  size  size  of  nominal  1750  machine  (gates) 

normalization  =  - 

factor  size  of  machine  E  (gates) 

The  microcode  and  the  main  storage  components  are  treated  in  a  similar 
fashion . 

As  an  example  of  the  use  of  this  method,  the  nominal  MIL-STD-1750 
machine  was  expected  to  contain  30,000  gates,  200,000  microcode  bits, 
ard  1,000,000  main  storage  bits.  Next,  suppose  that  Verification 
Method  A  was  used  on  a  large  machine  (E)  and  machine  (B)  contained 
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100, gates,  400,000  nicrccode  fcjts  and  10,000,000  main  storage 
tit':  Also  suppose  that  50  architectural  discrepancies  were 
pro  -,ted.  Of  these,  30  were  logic-related,  15  were 
cic  Je-related,  and  5  were  main  stcrage-related.  The  projected 
number  of  architectural  discrepancies  for  Verification  Method  A  for  a 
ncmin.'^  MIL-STD-1750  machine  becomes: 


30,000  gates  (1750) 

EISC££P(1750)  =  30  logic  DISCFEP(E)  X - 

100,000  gates  (B) 

200,000  hits  (1750) 

+  15  microcode  CISCFEF  (E)  x - 

400,000  tits  (B) 

1,000,000  bits  (1750) 

+  5  main  storage  DISCPEF(B)  x  - - - 

10,000,000  bits  (B) 

=  30  X  0.3  +  15  X  0.5  +  5  X  0.1 
=  17 


Verification  Method  A  would  he  expected  to  miss  17  (not  50) 
architectural  discrepa ncies  when  applied  tc  a  machine  of  the  nominal 
size  expected  to  be  used  in  MIL-STE-1750  applications. 

Ncrmalization  has  prevented  an  over-estimation  of  the  total  number  of 
architectural  discrepancies  for  Verification  Method  A  simply  because 
it  was  used  on  a  large  machine  (B)  . 
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3C.0  APPENDIX  C  -  PfCGRAMS  FCP  ES^I  MATING  TC^AL  NUnEER  GF 

AFCHITFCTURAL  DISCREPANCIES 


[1  ] 
121 
t31 

14] 

[5] 

[61 

t7] 

[0] 

191 

[101 

(111 

(121 

tl31 

(141 

(151 

(161 

(171 

(101 

(191 

(201 

(211 

(221 

(231 

(241 

(251 

(261 

(271 

(201 

(291 

(301 

(311 


vESTIMATEiaiv 
V  ESTIMATE 

A  ESTIMATE  C  (TOTAL  NUMBER  OF  EC’S)  BY  SUCCESSIUE  APPROXIMATION 
n  (0)  INITIALIZATION  ---  C«/  -  200  CLO  •  TOTAL  NUMBER  OF  KNOUN  EC'S 

«  (1)  ESTIMATE  Cl  MIDPOINT  OF  CHI  AND  CLO 

n  (2)  FIT  CURUE  TO  DATA  (USING  ESTIMATE  OF  C) 
n  (Z)  FIT  CURVES  USING  C*1  AND  C-i 

n  (A)  IF  SQUARED  ERROR  OF  CURVE  FOR  C  i  SQUARED  ERROR  OF  CURVES  FOR 
n  C^l  AND  C-1  THEN  C  IS  BEST  ESTIMATE t  STOP 

•  (5)  IF  SQUARED  ERROR  OF  CURVE  FOR  C-1  <  SQUARED  ERROR  OF 

n  CURVES  FOR  C  AND  C*l  THEN  CHI  -  C*lt  GO  TO  (1) 

•  (6)  IF  SQUARED  ERROR  OF  CURVE  FOR  C*i  <  SQUARED  ERROR  OF 

"  CURVES  FOR  C  AND  C-1  THEN  CLO  •  C-ll  GO  TO  (1) 

> 

Q«-'  I 

a«-'  A-7  ENGINEERING  CHANGES' 

□  4-'  I 

0^' MONTHS  SINCE  FIRST  SHIPt  '»  3  0  TX(il0J 
0*-' CUMULATIVE  EC" St  ',3  0  Ty(»101 

a*- 'MONTHS  SINCE  FIRST  SHIPt  ',3  0  TX(10*»171 
Q*-' CUMULATIVE  EC" St  ',3  0  Ty'(10^il71 

a«.'  ' 

CHI^200 
CL0*-n  (fV)-ii 

U*-' ESTIMATE  OF  C  SQUARED  ERROR  ' 

LOOPltCMIDH  (CHI*CL0)*2 
CMl*-CMID-l 
CM2*-CMID*i 
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C  32 1  C*-Ct1ID 

(331  X  FIT  y 

1 34  1  SQt1ID*-SOEKR 

(351  □«-  7  0  25  5  JCHID»SOrtID 

1 36 1  C*-CAri 

(371  •*(C>V'(  (fn-ll)/(.0 

(361  C«-C^0.1 

(391  LOtX  FIT  Y 
( 40  1  SQni*-SQERR 
(411  C*-CM2 
(421  X  FIT  Y 
( 43 1  SQn2*-SQERR 

(441  ■►(  lSQMIDiS0Hi)^iSQHIDiSQM2)  't/FIHISH 

(451  ■HSQHID>SOMi)/Ll 

(461  CLO*-CtH 

(471  -^LOOPi 

(481  LUCHI*-Cn2 

(491  •*LOOPi 

(501  FIHISHiU*-'  ' 

(511  U*-*  TOTAL  PROJECTED  ARCHITEC1URAL  DISCREPANCIES* 

(521  a*- •  (SQUARED  ERROR  TECHNIQUE)  •  '#TC«ID 

(531  D^'  • 

(541  ' 

(551  □♦•  •  • 
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vFiTiaiv 

V  X  FIT  y 

[1]  R 

[21  H  FIT  EXPONENTIAL  CURUE  TO  DATA  (C  -  Y)t 
[31  R  (0)  LINEARIZE  DATAt  LN  iC  -  Y) 

141  R  (1)  CONFUTE  LINEAR  LEAST  SQUARED  ESTIMATE  FOR  LINEARIZED  DATA 
[51  R  (2)  SLOPE  OF  RESULTING  LINE  IS  EXPONENT  FOR  EXPONENTIAL 
[61  R  (3)  COMPUTE  SQUARED  ERROR  BETUEEN  DATA  AND  EXPONENTIAL  CURUE 
[71  R 

[01  z«-«<c-n 

[91  MZ*-MEAN  Z 
[101  MX*-MEAN  X 
[111  UARZ*‘UAR  Z 
[121  UARX*-UAR  X 
[131  STDUZ*-UARZ»Q.S 
[141  STDUX*-UARX»(i,S 
[  15 1  CROSSSUM*-*/ (Z«X) 

[161  R 

[171  R  COMPUTE  R»  THE  CORRELATION  COEFFICIENT 
[101  R 

[  19 1  R^'UCROSSSUM^i  tZ) )  -  (MZ»MX)  )*iSTDUZ»STDUX) 

[201  SLOPEZ*-(R*STDUZ)*STDUX 
[211  R 

[221  R  FiC*SLOPEZ)  IS  THE  EQUATION  OF  THE  CUMULATIUE  EC  CURUE 
[231  R 

[241  SQERR*-*/i(.Y~{C  F  SL0PEZ't'i»2'i 

7 

[  □  17 

V  Z*‘C  F  A 

111 

[21 

7 


C-3 


6176n5A 


FINAL  BEPOFI  Fetruary  29, 


vvrtfflDJv 

V  2*-VAR  X 
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(31  K 

(41  Z*-((*y(X*2))*(fX))-((HEAN  X)»2) 
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0  APPENDIX  C  -  ESTIMATES  CF  TOTAL  APC HIT FCTDB AL  DISCFEPANCIES 


P-52D  SPN/GEANS  ENGINEERING  CHANGES 


MCNTHS  SINCE  FIHST  SHIP:  0  1  2  3  *1  5  6 
COMDLAIIVE  ECs;  0  15  17  19  20  25  26 


ESTIMATE  OF  C 
113 
70 
4-6 
37 
32 
29 
28 
27 


SQUARED  ERROR 
320. 35137 
272-04194 
207.97311 
140.82952 
91.88000 
54.96710 
42.76363 
34.35369 


TOTAL  PROJECTED  ARCHITECTOSAL  CISCFEE ANCIES 
(SQUARED  ERROR  TECHNIQUE)  =  27 


E-52D  SPN/GEANS  ENGINEERING  CHANGES 


SHIP:  0123456 

0  15  17  19  20  25  26 


MCNTHS  SINCE  FIRST 
COMOLAIIVE  ECs: 

ESTIMATE  OF  C 
113 
70 
48 
37 
32 

29 

30 


CORREIATICN  CCEFFICIENT 
-.91590 
-.92712 
-.94218 
-.95714 
-.96556 
-.96672 
-.96736 


TOTAL  PROJECTED  ARCHITECTURAL  DISCFEPANCIES 
(COFFEIATION  COEFFICIENT  TECHNIQUE)  =  30 
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A-7  ENGINEEFING  CKAEGES 


MONTHS  SINCE  FIRST 

SHIP: 

0 

1 

2 

2 

4 

5 

6 

7 

8 

9 

10 

11 

12 

CUMULATIVE  ECs; 

0 

0 

2 

2 

2 

6 

9 

9 

11 

12 

13 

13 

14 

MONTHS  SINCE  FIRST 

SHIP: 

13 

14 

15 

16 

17 

18 

19 

20 

2  1 

22 

23 

24 

25 

CUMULATIVE  ECs: 

14 

15 

15 

16 

17 

17 

20 

20 

20 

22 

22 

22 

22 

MONTHS  SINCE  FIRST 

SHIP: 

26 

27 

28 

29 

30 

31 

32 

33 

34 

CUMULATIVE  ECs: 

22 

22 

22 

22 

22 

22 

24 

24 

24 

ESTIMATE  01  C 
112 
68 
46 
35 
30 

27 

28 
29 


SQOAEED  EBEOP 
431.75694 
323.24758 
196.48119 
91.16330 
46.95553 
59.01848 
46.65670 
44.04095 


TOTAl  PEOJECTED  AECHITECTDP A1  tISCFEE AKC3ES 
(SQOAEED  EEEOE  TECHNIQUE)  =  29 


A-7  ENGINEEPING  CHANGES 


MONTHS  SINCE  FIRST 

SHIP; 

0 

1 

2 

3 

4 

5 

6 

7 

6 

9 

10 

11 

12 

CUMULATIVE 

ECs: 

0 

0 

2 

2 

2 

6 

9 

9 

11 

12 

13 

13 

14 

MONTHS  SINCE  FIRST 

SHIP: 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 

23 

24 

25 

CUMULATIVE 

ECs: 

14 

15 

15 

16 

17 

17 

20 

20 

20 

22 

22 

22 

22 

MONTHS  SINCE  FIRST 

SHIP: 

26 

27 

28 

29 

30 

31 

32 

33 

34 

CUMULATIVE 

ECs: 

22 

22 

22 

22 

22 

22 

24 

24 

24 

ESTIMATE  Of  C 
112 
68 
46 
35 
30 

27 

28 


COPFEIATICN  COEFFICIENT 
-. 95859 
-.96407 
-.97126 
-.97852 
-.98295 
-.98381 
-.98411 


TOTAL  PEOJECTED  AECHITECTDEAl  DISCFEP ANCIES 
(SQOAEED  EEEOE  TECHNIQUE)  =  29 
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S/370  MODEL  ENGINEEBIliG  CFANGES 


MOUTHS  SINCE  FIRST  SHIP: 
CDMDLAIIVE  ECs: 

MONTHS  SINCE  FIRST  SHIP: 
CDHOIATIVE  ECt.; 


2  3  U  5  6  7  e  9  10  11  12 
8  11  m  19  21  26  29  30  34  35  35 


13  14  15  16 
37  41  42  44 


ESTIMATE  OF 
122 
83 
64 
73 
68 


SQUARED  ERROR 
97. 82478 
33.64729 
19.76986 
19.42355 
16. 81260 


TOTAL  PROJECTED  ARCHITECTURAL  DISCREP AKCIES 
(SQUARED  ERROR  TECHNIQUE)  =  68 


S/370  MODEL  ENGINEERING  CHANGES 


HCNTHS  SINCE  FIRST  SHIP: 
CUMULATIVE  ECs: 

MCNTHS  SINCE  FIRST  SHIP: 
CUMULATIVE  ECs; 


4  5 
14  19 


8  S  10  11  12 
29  30  34  35  55 


13  14  15  16 
37  41  42  44 


ESTIMATE  OF 
122 
83 
64 
73 
68 
66 


COEEELATICN  CCEFFICIEN' 
-,99267 
-.99538 
-.99637 
-.99615 
-.99639 
-.99641 


TOTAL  PROJECTED  ARCHITECTURAL  DISCREPANCIES 
(CORRELATION  COEFFICIENT  TECHNIQUE)  =  66 


I 

I 
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50.0  APFESDII  E  ;  CERTIFICATION  INTER F AC J  tCCUMENT 

50.1  DESCRIPTION 

Tbe  Certification  Interface  Document  serves  as  a  means  of 
ccmmun ication  between  SEAPAC  and  the  vendor  for  the  certification 
process.  This  document  contains  detailed  information  of  the  following 
type: 

•  Certification  System  Hardware  Configuration 

•  Certification  Scenario 

-  Schedule  of  Events 

•  Physical  Resources  Available  to  the  Vendor 

-  Power 

-  Space 
Cooling 

-  Access  Times 

•  Vendor  Provided  Hardware 

MIL-STD- 1750  Computer  to  be  Tested 
aiL-STD-1553  I/C  Channel 

-  Cables 

•  Vendor  Provided  Software 

-  I/O  Subroutine  Descriptions 

•  SEAPAC  Provided  Software 

-  Support  Software 

-  Certification  Program  Source  (two  -  AVP  and  Random) 

-  Bootstrap  Program  Source 

•  Vendor  Certification  Personnel  Reouirements 
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SEAFAC  Certification  Perscnnel 
Observer 
Technician 
-  Coordinator 
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